Kernel analysis of deep networks

Journal of Machine Learning Research

Volumn 12, Issue , 2011, Pages 2563-2581

  Montavon, Grégoire ^a   Braun, Mikio L ^a   Müller, Klaus Robert ^a,b  

^a TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Deep networks; Kernel principal component analysis; Representations

Indexed keywords

COMMON KNOWLEDGE; DEEP NETWORKS; KERNEL PRINCIPAL COMPONENT ANALYSIS; LAYER-WISE; LEARNING PROBLEM; NETWORK CONTROL; REPRESENTATIONS;

KNOWLEDGE REPRESENTATION;

PRINCIPAL COMPONENT ANALYSIS;

EID: 80555140085     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (38)

1. Gaston Baudat and Fatiha Anouar. Generalized discriminant analysis using a kernel approach. Neural Computation, 12(10):2385-2404, 2000.
2. Yoshua Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1):1-127, 2009.
3. Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems 19, pages 153-160, 2006.
4. Christopher M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1996.
5. Leon Bottou. Stochastic gradient learning in neural networks. In Proceedings of Neuro-Nîmes, 1991.
6. Olivier Bousquet, Stéphane Boucheron, and Gabor Lugosi. Introduction to statistical learning theory. In Olivier Bousquet, Ulrike von Luxburg, and Gunnar Rätsch, editors, Advanced Lectures on Machine Learning, volume 3176, pages 169-207. Springer, 2004. (Pubitemid 39741632)
7. Mikio L. Braun. Accurate bounds for the eigenvalues of the kernel matrix. Journal of Machine Learning Research, 7:2303-2328, 2006. (Pubitemid 44708008)
8. Mikio L. Braun, Joachim Buhmann, and Klaus-Robert Müller. On relevant dimensions in kernel feature spaces. Journal of Machine Learning Research, 9:1875-1908, 2008.
9. Rich Caruana. Multitask learning. Machine Learning, 28(1):41-75, 1997. (Pubitemid 127507169)
10. Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. In Advances in Neural Information Processing Systems 22, pages 342-350, 2009.
11. Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, pages 160-167, 2008.
12. Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20(3):273-297, 1995.
13. Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11:625-660, 2010.
14. Ian Goodfellow, Quoc Le, Andrew Saxe, and Andrew Y. Ng. Measuring invariances in deep networks. In Advances in Neural Information Processing Systems 22, pages 646-654, 2009.
15. Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18(7):1527-1554, 2006. (Pubitemid 44024729)
16. David H. Hubel and Torsten N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of Physiology, 160:106-154, January 1962.
17. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
18. Kevin J. Lang, Alex H. Waibel, and Geoffrey E. Hinton. A time-delay neural network architecture for isolated word recognition. Neural Networks, 3(1):23-43, 1990.
19. Hugo Larochelle, Yoshua Bengio, Jérôme Louradour, and Pascal Lamblin. Exploring strategies for training deep neural networks. Journal of Machine Learning Research, 10:1-40, 2009.
20. Yann LeCun. Generalization and network design strategies. In Connectionism in Perspective. Elsevier, 1989. An extended version was published as a technical report of the University of Toronto.
21. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1):2278-2324, 1998.
22. Sebastian Mika, Gunnar Rätsch, Jason Weston, Bernhard Schölkopf, and Klaus-Robert Müller. Fisher discriminant analysis with kernels. In Neural Networks for Signal Processing IX, 1999. Proceedings of the 1999 IEEE Signal Processing Society Workshop, pages 41-48, 1999. (Pubitemid 30510358)
23. Sebastian Mika, Gunnar Rätsch, Jason Weston, Bernhard Schölkopf, Alex J. Smola, and Klaus-Robert Müller. Learning discriminative and invariant nonlinear features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(5):623-628, 2003.
24. Grégoire Montavon, Mikio Braun, and Klaus-Robert Müller. Layer-wise analysis of deep networks with Gaussian kernels. In Advances in Neural Information Processing Systems 23, pages 1678-1686, 2010.
25. Klaus-Robert Müller, Sebastian Mika, Gunnar Rätsch, Koji Tsuda, and Bernhard Schölkopf. An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks, 12(2):181-202, 2001.
26. Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning, pages 807-814, 2010.
27. Genevieve B. Orr and Klaus-Robert Müller, editors. Neural Networks: Tricks of the Trade, volume 1524 of Lecture Notes in Computer Science, 1998. Springer. This book is an outgrowth of a 1996 NIPS workshop.
28. Dario L. Ringach. Spatial structure and symmetry of simple-cell receptive fields in macaque primary visual cortex. The Journal of Neurophysiology, 88(1):455-463, 2002. (Pubitemid 34755364)
29. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, 323(6088):533-536, 1986. (Pubitemid 16025374)
30. Ruslan Salakhutdinov and Geoffrey Hinton. Learning a nonlinear embedding by preserving class neighbourhood structure. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 11, 2007.
31. Bernhard Schölkopf and Alexander J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, 2002.
32. Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10(5):1299-1319, 1998. (Pubitemid 128463674)
33. Bernhard Schölkopf, Sebastian Mika, Chris J. C. Burges, Philipp Knirsch, Klaus-Robert Müller, Gunnar Rätsch, and Alex J. Smola. Input space versus feature space in kernel-based methods. IEEE Transactions on Neural Networks, 10(5):1000-1017, 1999.
34. Thomas Serre, Lior Wolf, and Tomaso Poggio. Object recognition with features inspired by visual cortex. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2, pages 994-1000, 2005.
35. Steve Smale, Lorenzo Rosasco, Jack Bouvrie, Andrea Caponnetto, and Tomaso Poggio. Mathematics of the neural response. Foundations of Computational Mathematics, 10(1):67-91, 2010.
36. Alex J. Smola, Bernhard Schölkopf, and Klaus-Robert Müller. The connection between regularization operators and support vector kernels. Neural Networks, 11(4):637-649, 1998. (Pubitemid 28400264)
37. Jason Weston, Frédéric Ratle, and Ronan Collobert. Deep learning via semi-supervised embedding. In Proceedings of the 25th International Conference on Machine Learning, pages 1168-1175, 2008.
38. Andre Wibisono, Jake Bouvrie, Lorenzo Rosasco, and Tomaso Poggio. Learning and invariance in a family of hierarchical kernels. Technical report, Massachusetts Institute of Technology, 2010.