Exploring strategies for training deep neural networks

Journal of Machine Learning Research

Volumn 10, Issue , 2009, Pages 1-40

  Larochelle, Hugo ^a   Bengio, Yoshua ^a   Louradour, Jérôme ^a   Lamblin, Pascal ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

Artificial neural networks; Autoassociators; Deep belief networks; Restricted Boltzmann machines; Unsupervised learning

Indexed keywords

ALGORITHMS; BACKPROPAGATION; BAYESIAN NETWORKS; LEARNING ALGORITHMS; NETWORK LAYERS; OPTIMIZATION; UNSUPERVISED LEARNING;

ARTIFICIAL NEURAL NETWORKS; AUTOASSOCIATORS; DEEP BELIEF NETWORKS; DISTRIBUTED REPRESENTATIONS; GENERATIVE MODELS; GRADIENT ESTIMATORS; GRADIENT-BASED OPTIMIZATIONS; HIGH-LEVEL ABSTRACTIONS; LAYER WISE; LEARNING PROCEDURES; LOCAL MINIMUMS; MULTI-LAYER NEURAL NETWORKS; NON-LINEAR; NON-LINEARITIES; ON-LINE VERSIONS; OPTIMIZATION PROBLEMS; RESTRICTED BOLTZMANN MACHINES; TRAINING ALGORITHMS; UNSUPERVISED TRAININGS;

NEURAL NETWORKS;

EID: 59449087310     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (61)

1. Rie Kubota Ando and Tong Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817-1853, 2005.
2. Peter Auer, Mark Herbster, and Manfred K. Warmuth. Exponentially many local minima for single neurons. In M. Mozer, D. S. Touretzky, and M. Perrone, editors, Advances in Neural Information Processing System 8, pages 315-322. MIT Press, Cambridge, MA, 1996.
3. Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 2:53-58, 1989.
4. Anthony J. Bell and Terrence J. Sejnowski. An information maximisation approach to blind separation and blind deconvolution. Neural Computation, 7(6):1129-1159, 1995.
5. Yoshua Bengio. Learning deep architectures for AI. Technical Report 1312, Université de Montréal, dept. IRO, 2007.
6. Yoshua Bengio and Olivier Delalleau. Justifying and generalizing contrastive divergence. Technical Report 1311, Dept. IRO, Université de Montréal, 2007.
7. Yoshua Bengio and Yann Le Cun. Scaling learning algorithms towards AI. In L. Bottou, O. Chapelle, D. DeCoste, and J. Weston, editors, Large Scale Kernel Machines. MIT Press, 2007.
8. Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. The curse of highly variable functions for local kernel machines. In Y. Weiss, B. Schölkopf, and J. Piatt, editors, Advances in Neural Information Processing Systems 18, pages 107-114. MIT Press, Cambridge, MA, 2006.
9. Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 153-160. MIT Press, 2007.
10. Guillaume Bouchard and Bill Triggs. The tradeoff between generative and discriminative classifiers. In IASC International Symposium on Computational Statistics (COMPSTAT), pages 721-728, Prague, August 2004. URL http://lear.inrialpes.fr/pubs/2004/BT04.
11. Miguel A. Carreira-Perpiñan and Geoffrey E. Hinton. On contrastive divergence learning. In Robert G. Cowell and Zoubin Ghahramani, editors, Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, Jan 6-8, 2005, Savannah Hotel, Barbados, pages 33-40. Society for Artificial Intelligence and Statistics, 2005.
12. Hsin Chen and Alan F. Murray. A continuous restricted Boltzmann machine with an implementable training algorithm. IEE Proceedings of Vision, Image and Signal Processing, 150(3): 153-158, 2003.
13. Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the Twenty-fifth International Conference on Machine Learning (ICML 2008), pages 160-167, 2008. URL http://www.kyb.tuebingen.mpg.de/bs/ people/weston/papers/unified\-nlp.pdf.
14. Pierre Comon. Independent component analysis - a new concept? Signal Processing, 36:287-314, 1994.
15. Garrison W. Cottrell, Paul Munro, and David Zipser. Learning internal representations from grayscale images: An example of extensional programming. In Ninth Annual Conference of the Cognitive Science Society, pages 462-473, Seattle 1987, 1987. Lawrence Erlbaum, Hillsdale.
16. Peter Dayan, Geoffrey Hinton, Radford Neal, and Rich Zemel. The Helmholtz machine. Neural Computation, 7:889-904, 1995.
17. David DeMers and Garrison W. Cottrell. Non-linear dimensionality reduction. In C.L. Giles, S.J. Hanson, and J.D. Cowan, editors, Advances in Neural Information Processing Systems 5, pages 580-587, San Mateo CA, 1993. Morgan Kaufmann.
18. Scott E. Fahlman and Christian Lebiere. The cascade-correlation learning architecture. In D.S. Touretzky, editor, Advances in Neural Information Processing Systems 2, pages 524-532, Denver, CO, 1990. Morgan Kaufmann, San Mateo.
19. Ildiko E. Frank and Jerome H. Friedman. A statistical view of some chemometrics regression tools. Technometrics, 35(2): 109-148, 1993.
20. Brendan J. Frey. Graphical models for machine learning and digital communication. MIT Press, 1998.
21. Kenji Fukumizu and Shun-ichi Amari. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Networks, 13(3):317-327, 2000.
22. Raia Hadsell, Ayse Erkan, Pierre Sermanet, Marco Scoffier, Urs Muller, and Yann LeCun. Deep belief net learning in a long-range vision system for autonomous off-road driving. In Proc. Intelligent Robots and Systems (IROS'08), 2008. URL http://www.es.nyu.edu/- raia/docs/iros08-farod.pdf.
23. Johan Håstad. Almost optimal lower bounds for small depth circuits. In Proceedings of the 18th annual ACM Symposium on Theory of Computing, pages 6-20, Berkeley, California, 1986. ACM Press.
24. Johan Hastad and M. Goldmann. On the power of small-depth threshold circuits. Computational Complexity, 1:113-129, 1991.
25. Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14:1771-1800, 2002.
26. Geoffrey E. Hinton. Connectionist learning procedures. Artificial Intelligence, 40:185-234, 1989.
27. Geoffrey E. Hinton. To recognize shapes, first learn to generate images. Technical Report UTML TR 2006-003, University of Toronto, 2006.
28. Geoffrey E. Hinton and Ruslan R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507, July 2006.
29. Geoffrey E. Hinton, Peter Dayan, Brendan J. Frey, and Radford M. Neal. The wake-sleep algorithm for unsupervised neural networks. Science, 268:1558-1161, 1995.
30. Goeffrey E. Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554, 2006.
31. Alex Holub and Pietro Perona. A discriminative framework for modelling object classes. In CVPR '05: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, pages 664-671, Washington, DC, USA, 2005. IEEE Computer Society. ISBN 0-7695-2372-2. doi: http://dx.doi.org/10.1109/CVPR.2005.25.
32. Kurt Hornik, Maxwell Stinchcombe, and Haibert White. Multilayer feedforward networks are universal approximators. Neural Networks, 2:359-366, 1989.
33. Tommi S. Jaakkola and David Haussler. Exploiting generative models in discriminative classifiers. In M.S. Kearns, S.A. Solla, and D.A. Cohn, editors, Advances in Neural Information Processing Systems 11. MIT Press, Cambridge, MA, 1999.
34. Tony Jebara. Machine Learning: Discriminative and Generative (The Kluwer International Series in Engineering and Computer Science). Springer, December 2003. ISBN 1402076479. URL http://www.amazon.ca/exec/obidos/redirect? tag=citeulike09-20&path=ASIN/1402076479.
35. Christian Jutten and Jeanny Herault. Blind separation of sources, part I: an adaptive algorithm based on neuromimetic architecture. Signal Processing, 24:1-10, 1991.
36. Hugo Larochelle and Yoshua Bengio. Classification using discriminative restricted boltzmann machines. In Andrew McCallum and Sam Roweis, editors, Proceedings of the 25th Annual International Conference on Machine Learning (ICML 2008), pages 536-543. Omnipress, 2008.
37. Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. In Zoubin Ghahramani, editor, Twenty-fourth International Conference on Machine Learning (ICML 2007), pages 473-480. Omnipress, 2007. URL http://www.machinelearning.org/proceedings/icm12007/papers/ 331.pdf.
38. Julia A. Lasserre, Christopher M. Bishop, and Thomas P. Minka. Principled hybrids of generative and discriminative models. In CVPR '06: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 87-94, Washington, DC, USA, 2006. IEEE Computer Society. ISBN 0-7695-2597-0. doi: http://dx.doi.org/10.1109/CVPR.2006.227.
39. Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, November 1998.
40. Régis Lengellé and Thierry Denoeux. Training MLPs layer by layer using an objective function for internal representations. Neural Networks, 9:83-97, 1996.
41. Javier R. Movellan, Paul Mineiro, and R. J. Williams. A monte-carlo EM approach for partially observable diffusion processes: theory and applications to neural networks. Neural Computation, 14:1501-1544,2002.
42. Radford M. Neal. Connectionist learning of belief networks. Artificial Intelligence, 56:71-113, 1992.
43. Andrew Y Ng and Michael I. Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In NIPS, pages 841-848, 2001.
44. Simon Osindero and Geoffrey E. Hinton. Modeling image patches with a directed hierarchy of markov random field. In Neural Information Processing Systems Conference (NIPS) 20, 2008.
45. Marc'Aurelio Ranzato, Fu-Jie Huang, Y-Lan Boureau, and Yann LeCun. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In Proc. Computer Vision and Pattern Recognition Conference (CVPR'07). IEEE Press, 2007a.
46. Marc'Aurelio Ranzato, Christopher Poultney, Sumit Chopra, and Yann LeCun. Efficient learning of sparse representations with an energy-based model. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19. MIT Press, 2007b.
47. Marc'Aurelio Ranzato, Y-Lan Boureau, and Yann LeCun. Sparse feature learning for deep belief networks. In J.C. Piatt, D. Koller, Y Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20. MIT Press, Cambridge, MA, 2008. URL http://www.es.nyu.edu/-ranzato/publications/ ranzato-nips07.pdf.
48. Ruslan Salakhutdinov and Geoffrey Hinton. Using deep belief nets to learn covariance kernels for gaussian processes. In J. C. Piatt, D. Koller, Y Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20. MIT Press, Cambridge, MA, 2008. URL http://www.csri.utoronto.ca/~hinton/ absps/dbngp.pdf.
49. Ruslan Salakhutdinov and Geoffrey Hinton. Semantic hashing. In Proceedings of the 2007 Workshop on Information Retrieval and applications of Graphical Models (SIGIR 2007), Amsterdam, 2007a. Elsevier.
50. Ruslan Salakhutdinov and Geoffrey Hinton. Learning a nonlinear embedding by preserving class neighbourhood structure. In Proceedings of AISTATS 2007, San Juan, Porto Rico, 2007b. Omnipress.
51. Ruslan Salakhutdinov and Iain Murray. On the quantitative analysis of deep belief networks. In Proceedings of the International Conference on Machine Learning, volume 25, 2008.
52. Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey Hinton. Restricted boltzmann machines for collaborative filtering. In ICML '07: Proceedings of the 24th international conference on Machine learning, pages 791-798, New York, NY, USA, 2007. ACM.
53. Lawrence K. Saul, Tommi Jaakkola, and Michael I. Jordan. Mean field theory for sigmoid belief networks. Journal of Artificial Intelligence Research, 4:61-76, 1996.
54. Eric Saund. Dimensionality-reduction using connectionist networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(3):304-314, 1989.
55. Paul Smolensky. Information processing in dynamical systems: Foundations of harmony theory. In D. E. Rumelhart and J. L. McClelland, editors, Parallel Distributed Processing, volume 1, chapter 6, pages 194-281. MIT Press, Cambridge, 1986.
56. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Andrew McCallum and Sam Roweis, editors, Proceedings of the 25th Annual International Conference on Machine Learning (ICML 2008), pages 1096-1103. Omnipress, 2008. URL http://icml2008.es.helsinki.fi/papers/592.pdf.
57. Ingo Wegener. The Complexity of Boolean Functions. John Wiley & Sons, 1987.
58. Max Welling and Geoffrey E. Hinton. A new learning algorithm for mean field boltzmann machines. In ICANN '02: Proceedings of the International Conference on Artificial Neural Networks, pages 351-357, London, UK, 2002. Springer-Verlag. ISBN 3-540-44074-7.
59. Max Welling, Michal Rosen-Zvi, and Geoffrey E. Hinton. Exponential family harmoniums with an application to information retrieval. In L.K. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems 17. MIT Press, 2005.
60. Jason Weston, Frédéric Ratle, and Ronan Collobert. Deep learning via semi-supervised embedding. In Proceedings of the Twenty-fifth International Conference on Machine Learning (ICML 2008), 2008. URL http://www.kyb.tuebingen.mpg.de/bs/people/weston/papers/deep-embed.pdf.
61. Andrew Yao. Separating the polynomial-time hierarchy by oracles. In Proceedings of the 26th Annual IEEE Symposium on Foundations of Computer Science, pages 1-10, 1985.