Denoising autoencoder with modulated lateral connections learns invariant representations of natural images

3rd International Conference on Learning Representations, ICLR 2015 - Workshop Track Proceedings

Volumn , Issue , 2015, Pages

  Rasmus, Antti ^a   Raiko, Tapani ^a   Valpola, Harri ^b  

^a AALTO UNIVERSITY   (Finland)

^b ZenRobotics Ltd   (Finland)

Author keywords

[No Author keywords available]

Indexed keywords

DECODING; SIGNAL ENCODING;

AUTO ENCODERS; DE-NOISING; INVARIANT FEATURES; INVARIANT REPRESENTATION; LATERAL CONNECTIONS; NATURAL IMAGES; PROBABILITY MODELING; REAL-WORLD IMAGE;

LEARNING SYSTEMS;

EID: 85083953385     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (22)

1. Bengio, Y, Lamblin, P, Popovici, D, and Larochelle, H. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems (NIPS 2006), 2007.
2. Bengio, Y. Deep learning of representations: Looking forward. In Statistical Language and Speech Processing, pp. 1–37. Springer, 2013.
3. Bengio, Y, Yao, L, Alain, G, and Vincent, P. Generalized denoising auto-encoders as generative models. In Advances in Neural Information Processing Systems, pp. 899–907, 2013.
4. Ciresan, D. C, Meier, U, Gambardella, L. M, and Schmidhuber, J. Deep big simple neural nets for handwritten digit recogntion. Neural Computation, 22(12):3207–3220, 2010.
5. Dosovitskiy, A, Springenberg, J. T, Riedmiller, M, and Brox, T. Discriminative unsupervised feature learning with convolutional neural networks. In Ghahramani, Z, Welling, M, Cortes, C, Lawrence, N, and Weinberger, K (eds.), Advances in Neural Information Processing Systems (NIPS 2014), pp. 766–774. 2014.
6. Földiák, P. Learning invariance from transformation sequences. Neural Computation, 3:194–200, 1991.
7. Fukushima, K. Neural network model for a mechanism of pattern recognition unaffected by shift in position - Neocognitron. Trans. IECE, J62-A(10):658–665, 1979.
8. Goodfellow, I, Lee, H, Le, Q. V, Saxe, A, and Ng, A. Y. Measuring invariances in deep networks. In Advances in Neural Information Processing Systems (NIPS), pp. 646–654, 2009.
9. Hubel, D. H and Wiesel, T. N. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of physiology, 160(1):106, 1962.
10. Hyvärinen, A and Hoyer, P. Emergence of phase-and shift-invariant features by decomposition of natural images into independent feature subspaces. Neural computation, 12(7):1705–1720, 2000.
11. Kingma, D. P, Mohamed, S, Rezende, D. J, and Welling, M. Semi-supervised learning with deep generative models. In Ghahramani, Z, Welling, M, Cortes, C, Lawrence, N, and Weinberger, K (eds.), Advances in Neural Information Processing Systems (NIPS 2014), pp. 3581–3589. Curran Associates, Inc., 2014.
12. Krizhevsky, A and Hinton, G. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
13. Krizhevsky, A, Sutskever, I, and Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS 2012), pp. 1106–1114, 2012.
14. Memisevic, R. Gradient-based learning of higher-order image features. In 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1591–1598, November 2011.
15. Olshausen, B. A and Field, D. J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381:607–609, 1996.
16. Raiko, T, Valpola, H, and LeCun, Y. Deep learning made easier by linear transformations in percep-trons. In Lawrence, N. D and Girolami, M (eds.), AISTATS, volume 22 of JMLR Proceedings, pp. 924–932. JMLR.org, 2012.
17. Riesenhuber, M and Poggio, T. Hierarchical models of object recognition in cortex. Nature Neuroscience, 2(11):1019–1025, 1999.
18. Särelä, J and Valpola, H. Denoising source separation. Journal of Machine Learning Research, 6: 233–272, 2005.
19. Schmidhuber, J. Deep learning in neural networks: an overview. Neural Networks, 61:85–117, 2015.
20. Valpola, H. From neural PCA to deep unsupervised learning. In Advances in Independent Component Analysis and Learning Machines. Elsevier, 2015. Preprint available as arXiv:1411.7783.
21. Vincent, P, Larochelle, H, Bengio, Y, and Manzagol, P.-A. Extracting and composing robust features with denoising autoencoders. Technical Report 1316, Université de Montréal, dept. IRO, 2008.
22. Zeiler, M. D. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.