Learning with hidden variables

Current Opinion in Neurobiology

Volumn 35, Issue , 2015, Pages 110-118

  Roudi, Yasser ^a,b   Taylor, Graham ^c  

^a NORWEGIAN UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Norway)

^b NORDITA   (Denmark)

^c UNIVERSITY OF GUELPH   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

BACK PROPAGATION; BRAIN CORTEX; BRAIN NERVE CELL; CELL FUNCTION; DEEP BELIEF NETWORK; LEARNING; LEARNING ALGORITHM; MACHINE LEARNING; NERVE CELL NETWORK; NERVE CELL PLASTICITY; PRIORITY JOURNAL; REVIEW; ANIMAL; ARTIFICIAL NEURAL NETWORK; HUMAN; PHYSIOLOGY;

ANIMALS; CEREBRAL CORTEX; HUMANS; LEARNING; MACHINE LEARNING; NERVE NET; NEURAL NETWORKS (COMPUTER);

EID: 84939788643     PISSN: 09594388     EISSN: 18736882     Source Type: Journal    
DOI: 10.1016/j.conb.2015.07.006     Document Type: Review

References (81)

1. MacKay D.M. Towards an information-flow model of human behaviour. Br J Psychol 1956, 47:30-43.
2. Hubel D., Wiesel T. Shape and arrangement of columns in cat's striate cortex. J Physiol 1963, 165:559-568.
3. Marr D., Nishihara H.K. Representation and recognition of the spatial organization of three-dimensional shapes. Proc R Soc Lond B: Biol Sci 1978, 200:269-294.
4. Fukushima K. Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybernet 1980, 36:193-202.
5. Attneave F. Some informational aspects of visual perception. Psychol Rev 1954, 61:183.
6. Barlow HB: Possible principles underlying the transformations of sensory messages..
7. Rummelhart D. Learning representations by back-propagation errors. Nature 1986, 323:533-536.
8. Ackley D.H., Hinton G.E., Sejnowski T.J. A learning algorithm for Boltzmann machines. Cogn Sci 1985, 9:147-169.
9. Kappen H.J., Rodriguez F.B. Efficient learning in Boltzmann machines using linear response theory. Neurol Comput 1998, 10:1137-1156.
10. Tanaka T. Mean-field theory of Boltzmann machine learning. Phys Rev E 1998, 58:2302-2310.
11. Galland C.C. The limitations of deterministic Boltzmann machine learning. Network: Comput Neural Syst 1993, 4:355-379.
12. Dayan P., Hinton G.E., Neal R.M., Zemel R.S. The Helmholtz machine. Neural Comput 1995, 7:889-904.
13. Hinton G.E., Dayan P., Frey B.J., Neal R.M. The "wake-sleep" algorithm for unsupervised neural networks. Science 1995, 268:1158-1161.
14. Hinton G., Osindero S., Teh Y.-W. A fast learning algorithm for deep belief nets. Neural Comput 2006, 18:1527-1554.
15. Smolensky P. Information processing in dynamical systems: Foundations of harmony theory. Parallel Distributed Processing: vol 1: Foundations 1986, 194-281. MIT Press.
16. Hinton G.E. Training products of experts by minimizing contrastive divergence. Neural Comput 2002, 14:1771-1880.
17. Erhan D., Bengio Y., Courville A., Manzagol P.-A., Vincent P., Bengio S. Why does unsupervised pre-training help deep learning?. J Mach Learn Res 2010, 11:625-660.
18. Larochelle H., Erhan D., Courville A., Bergstra J., Bengio Y. An empirical evaluation of deep architectures on problems with many factors of variation. Proceedings of the 24th Annual International Conference on Machine Learning (ICML) 2007, 473-480.
19. Vincent P., Larochelle H., Bengio Y., Manzagol P.-A. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th Annual International Conference on Machine Learning (ICML) 2008, 1096-1103.
20. Ranzato M., Poultney C.S., Chopra S., LeCun Y. Efficient learning of sparse representations with an energy-based model. Advances in Neural Information Processing Systems (NIPS), vol 19 2007, 1137-1144.
21. Ranzato M., Boureau Y., LeCun Y. Sparse feature learning for deep belief networks. Advances in Neural Information Processing Systems (NIPS), vol 20 2008.
22. Kavukcuoglu K., Ranzato M., LeCun Y. Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition 2008, Tech. Rep., Computational and Biological Learning Lab, Courant Institute, NYU, cBLL-TR-2008-12-01.
23. Salakhutdinov R., Hinton G.E. Deep Boltzmann machines. International Conference on Artificial Intelligence and Statistics 2009, 448-455.
24. Srivastava N., Salakhutdinov R.R. Multimodal learning with deep Boltzmann machines. Advances in Neural Information Processing Systems 2012, 2222-2230.
25. Salakhutdinov R., Tenenbaum J.B., Torralba A. Learning with hierarchical-deep models. IEEE Trans Pattern Anal Mach Intell 2013, 35:1958-1971.
26. Goodfellow I., Mirza M., Courville A., Bengio Y. Multi-prediction deep Boltzmann machines. Advances in Neural Information Processing Systems 2013, 548-556.
27. LeCun Y., Bottou L., Bengio Y., Haffner P. Gradient-based learning applied to document recognition. Proc IEEE 1998, 86:2278-2324.
28. Krizhevsky A., Sutskever I., Hinton G. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, vol 25 2012, 1106-1114.
29. Karpathy A., Toderici G., Shetty S., Leung T., Sukthankar R., Li F.-F. Large-scale video classification with convolutional neural networks. CVPR 2014.
30. Simonyan K., Zisserman A. Two-stream convolutional networks for action recognition in videos. Advances in Neural Information Processing Systems 2014.
31. Mao J., Xu W., Yang Y., Wang J., Yuille A. Deep captioning with multimodal recurrent neural networks (m-RNN) 2014, http://arxiv.org/abs/1412.6632.
32. Xu K., Ba J., Kiros R., Courville A., Salakhutdinov R., Zemel R., Bengio Y. Show, attend and tell: Neural image caption generation with visual attention 2015, http://arxiv.org/abs/1502.03044.
33. Vinyals O., Toshev A., Bengio S., Erhan D. Show and tell: a neural image caption generator 2014, http://arxiv.org/abs/1411.4555.
34. Farabet C., Couprie C., Najman L., LeCun Y. Learning hierarchical features for scene labeling. IEEE Trans Pattern Anal Mach Intell 2013, 35:1915-1929.
35. Razavian A.S., Azizpour H., Sullivan J., Carlsson S. CNN features off-the-shelf: an astounding baseline for recognition. 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 2014, 512-519. IEEE.
36. Kavukcuoglu K., Ranzato M., LeCun Y. Fast inference in sparse coding algorithms with applications to object recognition 2010, http://arxiv.org/pdf/1010.3467.
37. Rifai S. Contractive auto-encoders: Explicit invariance during feature extraction. ICML 2011.
38. Bengio Y., Laufer E., Alain G., Yosinski J. Deep generative stochastic networks trainable by backprop. Proceedings of the 31st International Conference on Machine Learning (ICML-14) 2014, 226-234.
39. Rezende D.J., Mohamed S., Wierstra D. Stochastic backpropagation and approximate inference in deep generative models 2014, Tech. Rep. http://arxiv.org/abs/1401.4082.
40. Kingma D.P., Welling M. Auto-encoding variational Bayes. Proceedings of the International Conference on Learning Representations (ICLR) 2014.
41. Mnih A., Gregor K. Neural variational inference and learning in belief networks. ICML'2014 2014.
42. Hopfield J.J. Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci U S A 1982, 79:2554-2558.
43. Amit D.J., Gutfreund H., Sompolinsky H. Spin-glass models of neural networks. Phys Rev A 1985, 32:1007.
44. Mezard M., Parisi G., Virasoro M. Spin Glass Theory and Beyond 1987, World Scientific, Singapore.
45. Kühn R., Bös S., van Hemmen J.L. Statistical mechanics for networks of graded-response neurons. Phys Rev A 1991, 43:2084.
46. Treves A. Are spin-glass effects relevant to understanding realistic auto-associative networks?. J Phys A: Math Gen 1991, 24:2645.
47. Sutskever I., Hinton G.E. Learning multilevel distributed representations for high-dimensional sequences. Proceedings of the 11th International Conference on Artificial Intelligence and Statistics (AISTATS) 2007, 548-555.
48. Taylor G.W., Hinton G.E., Roweis S.T. Modeling human motion using binary latent variables. Advances in Neural Information Processing Systems 2006, 1345-1352.
49. Sutskever I., Hinton G.E., Taylor G.W. The recurrent temporal restricted Boltzmann machine. Advances in Neural Information Processing Systems 2009, 1601-1608.
50. Taylor G.W., Sigal L., Fleet D.J., Hinton G.E. Dynamical binary latent variable models for 3d human pose tracking. 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2010, 631-638. IEEE.
51. Mandel M.I., Pascanu R., Eck D., Bengio Y., Aiello L.M., Schifanella R., Menczer F. Contextual tag inference. ACM Trans Multimedia Comput Commun Appl 2011, 7:32.
52. Zeiler M.D., Taylor G.W., Sigal L., Matthews I., Fergus R. Facial expression transfer with input-output temporal restricted Boltzmann machines. Advances in Neural Information Processing Systems 2011, 1629-1637.
53. Roudi Y., Hertz J. Mean field theory for nonequilibrium network reconstruction. Phys Rev Lett 2011, 106:048702.
54. Zeng H.-L., Alava M., Mahmoudi H., Aurell E. Network inference using asynchronously updated kinetic Ising model. Phys Rev E 2011, 83:041135.
55. Mezard M, Sakellariou J: Exact mean-field inference in asymmetric kinetic Ising systems. J Stat Mech: Theory Exp.
56. Dunn B., Roudi Y. Learning and inference in a nonequilibrium Ising model with hidden nodes. Phys Rev E 2013, 87:022127.
57. Hertz J., Tyrcha J. Network inference with hidden nodes. Math Biosci Eng 2014, 11:149-156.
58. Bachschmid-Romano L., Opper M. Inferring hidden states in a random kinetic Ising model: replica analysis. J Stat Mech: Theory Exp 2014, 2014:P06013.
59. Battistin C, Hertz J, Tyrcha J, Roudi Y: Belief propagation and replicas for inference and learning in a kinetic Ising model with hidden spins. J Stat Mech P05021.
60. Hochreiter S., Bengio Y., Frasconi P., Schmidhuber J. Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-Term Dependencies 2001.
61. Martens J., Sutskever I. Learning recurrent neural networks with Hessian-free optimization. Proceedings of the 28th International Conference on Machine Learning (ICML-11) 2011, 1033-1040.
62. Hochreiter S., Schmidhuber J. Long short-term memory. Neural Comput 1997, 9:1735-1780.
63. Chung J., Gulcehre C., Cho K., Bengio Y. Empirical evaluation of gated recurrent neural networks on sequence modeling 2014, http://arxiv.org/abs/1412.3555.
64. Sutskever I., Martens J., Dahl G., Hinton G. On the importance of initialization and momentum in deep learning. Proceedings of the 30th International Conference on Machine Learning (ICML-13) 2013, 1139-1147.
65. Le Q.V., Jaitly N., Hinton G.E. A simple way to initialize recurrent networks of rectified linear units 2015, http://arxiv.org/abs/1504.00941.
66. Pascanu R., Mikolov T., Bengio Y. On the difficulty of training recurrent neural networks 2012, http://arxiv.org/abs/1211.5063.
67. Graves A., Mohamed A.-R., Hinton G. Speech recognition with deep recurrent neural networks. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2013, 6645-6649. IEEE.
68. Graves A. Generating sequences with recurrent neural networks 2013, http://arxiv.org/abs/1308.0850.
69. Sutskever I., Vinyals O., Le Q.V. Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems 2014, 3104-3112.
70. Mikolov T., Joulin A., Chopra S., Mathieu M., Ranzato M. Learning longer memory in recurrent neural networks 2014, http://arxiv.org/abs/1412.7753.
71. Hannun A., Case C., Casper J., Catanzaro B., Diamos G., Elsen E., Prenger R., Satheesh S., Sengupta S., Coates A., et al. Deepspeech: scaling up end-to-end speech recognition 2014, http://arxiv.org/abs/1412.5567.
72. O'Reilly R.C., Frank M.J. Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput 2006, 18:283-328.
73. Bengio Y., Lee D.-H., Bornschein J., Lin Z. Towards biologically plausible deep learning 2015, http://arxiv.org/abs/1502.04156.
74. Lee D.-H., Zhang S., Biard A., Bengio Y. Target propagation 2014, http://arxiv.org/abs/1412.7525.
75. Sountsov P., Miller P. Spiking neuron network Helmholtz machine. Front Comput Neurosci 2015, 9:46.
76. Treves A. Graded-response neurons and information encodings in autoassociative memories. Phys Rev A 1990, 42:2418.
77. Shriki O., Hansel D., Sompolinsky H. Rate models for conductance-based cortical neuronal networks. Neural Comput 2003, 15:1809-1841.
78. Roudi Y., Treves A. Localized activity profiles and storage capacity of rate-based autoassociative networks. Phys Rev E 2006, 73:061904.
79. Glorot X., Bordes A., Bengio Y. Deep sparse rectifier networks. Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, JMLR W&CP Volume, vol 15 2011, 315-323.
80. He K., Zhang X., Ren S., Sun J. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification 2015, http://arxiv.org/abs/1502.01852.
81. Hinton G.E., Srivastava N., Krizhevsky A., Sutskever I., Salakhutdinov R.R. Improving neural networks by preventing co-adaptation of feature detectors 2012, http://arxiv.org/abs/1207.0580.