From neural PCA to deep unsupervised learning

Advances in Independent Component Analysis and Learning Machines

Volumn , Issue , 2015, Pages 143-171

  Valpola, Harri ^a  

^a ZenRobotics Ltd   (Finland)

Author keywords

Deep learning; Denoising autoencoder; Denoising source separation; Invariant features; Ladder network; Unsupervised learning

Indexed keywords

EID: 85076816429     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1016/B978-0-12-802806-3.00008-7     Document Type: Chapter

References (26)

1. D.H. Hubel., T. Wiesel Receptive fields, binocular interaction, and functional architecture in the cat’s visual cortex J. Physiol. Lond. 160 1962 106-154.
2. K. Fukushima Neural network model for a mechanism of pattern recognition unaffected by shift in position - neocognitron Trans. IECE J62-A 10 1979 658-665.
3. J. Schmidhuber Deep learning in neural networks: an overview Neural Networks 61 2015 85-117.
4. G. Hinton., R. Salakhutdinov Reducing the dimensionality of data with neural networks Science 313 5786 2006 504-507.
5. G.E. Hinton., S. Osindero., Y.-W. Teh A fast learning algorithm for deep belief nets Neural Comput. 18 7 2006 1527-1554.
6. D.C. Ciresan., U. Meier., L.M. Gambardella., J. Schmidhuber Deep big simple neural nets for handwritten digit recognition Neural Comput. 22 2010 3207-3220.
7. A. Krizhevsky., I. Sutskever Hinton, ImageNet classification with deep convolutional neural networks Advances in Neural Information Processing Systems (NIPS 2012) 2012 1106-1114.
8. J. Särelä., H. Valpola Denoising source separation J. Mach. Learn. Res. 6 2005 233-272.
9. P. Vincent., L. Hugo., Y. Bengio Manzagol, Extracting and composing robust features with denoising autoencoders Proceedings of the 25th International Conferenceon Machine Learning (ICML '08) 2008 Helsinki Finland 1096-1103.
10. C. Bishop Latent variable models M. Jordan. Learning in Graphical Models 1999 MIT Press Cambridge, MA 371-403.
11. H. Valpola., M. Harva., J. Karhunen Hierarchical models of variance sources Signal Process. 84 2 2004 267-282.
12. K. Gregor., I. Danihelka., A. Mnih., C. Blundell., D. Wierstra Deep autoregressive networks, in: Proceedings of the 31st International Conference on Machine Learning,ICML’14 2014 Beijing China 1242-1250.
13. A. Ilin., H. Valpola On the effect of the form of the posterior approximation in variational learning of ICA models Proc. of the 4th Int. Symp. on Independent Component Analysis and Blind Signal Separation (ICA2003) 2003 Nara Japan 915-920.
14. E. Oja A simplified neuron model as a principal component analyzer J. Math. Biol. 15 1982 267-273.
15. E. Oja The nonlinear PCA learning rule in independent component analysis Neurocomputing 17 1 1997 25-46.
16. A. Hyvärinen., E. Oja A fast fixed-point algorithm for independent component analysis Neural Comput. 9 7 1997 1483-1492.
17. A. Hyvärinen Independent component analysis in the presence of Gaussian noise by maximizing joint likelihood Neurocomputing 22 1-3 1998 49-67.
18. H. Valpola., P. Pajunen Fast algorithms for Bayesian independent component analysis Proc. Int. Workshop on Independent Component Analysis and Blind Signal Separation (ICA2000) 2000 Helsinki Finland 233-237.
19. A.P. Dempster., N.M. Laird., D.B. Rubin Maximum likelihood from incomplete data via the EM algorithm J. R. Stat. Soc. Ser. B 39 1 1977 1-38.
20. M.S.C. Almeida., H. Valpola., J. Särelä Separation of nonlinear image mixtures by denoising source separation Proceedings of the 6th International Conference onIndependent Component Analysis and Blind Signal Separation, ICA 2006 2006 Charleston SC, USA 8-15.
21. Y. Bengio., É Thibodeau-Laufer., G. Alain., J. Yosinski Deep generative stochastic networks trainable by backprop Proceedings of the 31st International Conferenceon Machine Learning, ICML’14 2014 Beijing China 226-234.
22. B. Uria., I. Murray., H. Larochelle A deep and tractable density estimator Proceedings of the 31st International Conference on Machine Learning, ICML’14 2014 Beijing China 467-475.
23. A. Hyvärinen., P. Hoyer Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces Neural Comput. 12 7 2000 1705-1720.
24. T. Raiko., L. Yao., K. Cho., Y. Bengio Iterative neural autoregressive distribution estimator (NADE-k) Advances in Neural Information Processing Systems 27 (NIPS 2014) 2014 325-333.
25. A. Yli-Krekola A bio-inspired computational model of covert attention and learning (MA thesis) 2007 Helsinki University of Technology Finland.
26. G. Deco., E.T. Rolls A neurodynamical cortical model of visual attention and invariant object recognition Vision Res. 44 2004 621-642.