Zero-bias autoencoders and the benefits of co-adapting features

3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings

Volumn , Issue , 2015, Pages

  Konda, Kishore ^a   Memisevic, Roland ^b   Krueger, David ^b  

^a GOETHE UNIVERSITY   (Germany)

^b UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

CHEMICAL ACTIVATION;

ACTIVATION FUNCTIONS; DATA DISTRIBUTION; INTRINSIC DIMENSIONALITIES; NEGATIVE VALUES; NEW ACTIVATION FUNCTIONS; RECONSTRUCTION ERROR; SELECTION MECHANISM; TRAINING DATA;

LEARNING SYSTEMS;

EID: 85083951340     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (31)

1. Ba, Jimmy and Frey, Brendan. Adaptive dropout for training deep neural networks. In Advances in Neural Information Processing Systems, pp. 3084–3092, 2013.
2. Boche, Holger, Guillemard, Mijail, Kutyniok, Gitta, and Philipp, Friedrich. Signal recovery from thresholded frame measurements. In SPIE 8858, Wavelets and Sparsity XV, August 2013.
3. Coates, Adam, Lee, Honglak, and Ng, A. Y. An analysis of single-layer networks in unsupervised feature learning. In Artificial Intelligence and Statistics, 2011.
4. Courville, Aaron C, Bergstra, James, and Bengio, Yoshua. A spike and slab restricted boltzmann machine. In International Conference on Artificial Intelligence and Statistics, pp. 233–241, 2011.
5. Goroshin, Rotislav and LeCun, Yann. Saturating auto-encoders. In International Conference on Learning Representations (ICLR2013), April 2013.
6. Hinton, Geoffrey. A Practical Guide to Training Restricted Boltzmann Machines. Technical report, University of Toronto, 2010.
7. Hinton, Geoffrey E., Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, abs/1207.0580, 2012.
8. Hyvärinen, Aapo, Karhunen, Juha, and Oja, Erkki. Independent component analysis, volume 46. John Wiley & Sons, 2004.
9. Kamyshanska, Hanna and Memisevic, Roland. On autoencoder scoring. In Proceedings of the 30th International Conference on Machine Learning (ICML 2013), 2013.
10. Kavukcuoglu, Koray, Ranzato, Marc’Aurelio, and LeCun, Yann. Fast inference in sparse coding algorithms with applications to object recognition. arXiv preprint arXiv:1010.3467, 2010.
11. Konda, Kishore Reddy, Memisevic, Roland, and Michalski, Vincent. The role of spatio-temporal synchrony in the encoding of motion. In International Conference on Learning Representations (ICLR2014), 2014.
12. Kovacevic, J. and Chebira, A. An Introduction to Frames. Foundations and trends in signal processing. Now Publishers, 2008.
13. Krizhevsky, Alex and Hinton, Geoffrey. Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto, 2009.
14. Le, Quoc, Sarlos, Tamas, and Smola, Alex. Fastfood - approximating kernel expansions in loglinear time. In 30th International Conference on Machine Learning (ICML), 2013.
15. Le, Q.V., Zou, W.Y., Yeung, S.Y., and Ng, A.Y. Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. In CVPR, 2011.
16. Lee, Honglak, Ekanadham, Chaitanya, and Ng, Andrew. Sparse deep belief net model for visual area v2. In Advances in Neural Information Processing Systems 20, 2008.
17. Makhzani, Alireza and Frey, Brendan. k-sparse autoencoders. CoRR, abs/1312.5663, 2013.
18. Marszałek, Marcin, Laptev, Ivan, and Schmid, Cordelia. Actions in context. In IEEE Conference on Computer Vision & Pattern Recognition, 2009.
19. Martens, James, Chattopadhyay, Arkadev, Pitassi, Toniann, and Zemel, Richard. On the representational efficiency of restricted boltzmann machines. In Neural Information Processing Systems (NIPS) 2013, 2013.
20. Memisevic, Roland. Gradient-based learning of higher-order image features. In ICCV, 2011.
21. Memisevic, Roland and Hinton, Geoffrey. Unsupervised learning of image transformations. In CVPR, 2007.
22. Memisevic, Roland and Hinton, Geoffrey E. Learning to represent spatial transformations with factored higher-order boltzmann machines. Neural Computation, 22(6):1473–1492, June 2010. ISSN 0899-7667.
23. Olshausen, Bruno, Cadieu, Charles, Culpepper, Jack, and Warland, David. Bilinear models of natural images. In SPIE Proceedings: Human Vision Electronic Imaging XII, San Jose, 2007.
24. Ranzato, M, Huang, Fu Jie, Boureau, Y-L, and LeCun, Yann. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, pp. 1–8. IEEE, 2007.
25. Ranzato, Marc’Aurelio, Krizhevsky, Alex, and Hinton, Geoffrey E. Factored 3-Way Restricted Boltzmann Machines For Modeling Natural Images. In Artificial Intelligence and Statistics, 2010.
26. Razvan Pascanu, Guido Montufar, Yoshua Bengio. On the number of inference regions of deep feed forward networks with piece-wise linear activations. CoRR, arXiv:1312.6098, 2014.
27. Rifai, Salah, Vincent, Pascal, Muller, Xavier, Glorot, Xavier, and Bengio, Yoshua. Contractive Auto-Encoders: Explicit Invariance During Feature Extraction. In ICML, 2011.
28. Rozell, Christopher J, Johnson, Don H, Baraniuk, Richard G, and Olshausen, Bruno A. Sparse coding via thresholding and local competition in neural circuits. Neural computation, 20(10):2526–2563, 2008.
29. Saxe, Andrew, Koh, Pang Wei, Chen, Zhenghao, Bhand, Maneesh, Suresh, Bipin, and Ng, Andrew. On random weights and unsupervised feature learning. In Proceedings of the 28th International Conference on Machine Learning, 2011.
30. Taylor, Graham W., Fergus, Rob, LeCun, Yann, and Bregler, Christoph. Convolutional learning of spatio-temporal features. In Proceedings of the 11th European conference on Computer vision: Part VI, ECCV’10, 2010.
31. Vincent, Pascal, Larochelle, Hugo, Bengio, Yoshua, and Manzagol, Pierre-Antoine. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, 2008.