Knowledge matters: Importance of prior information for optimization

Journal of Machine Learning Research

Volumn 17, Issue , 2016, Pages

  Gülçehre, Çǧlar ^a   Bengio, Yoshua ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

Curriculum learning; Deep learning; Evolution of culture; Neural networks; Optimization; Training with hints

Indexed keywords

ARTIFICIAL INTELLIGENCE; CURRICULA; LEARNING SYSTEMS; NETWORK ARCHITECTURE; NEURAL NETWORKS; OPTIMIZATION;

CULTURAL LEARNING; DEEP LEARNING; GENERALIZATION ERROR; INTERMEDIATE CONCEPT; INTERMEDIATE LEVEL; POSITIVE EVIDENCE; PRIOR INFORMATION; SUPERVISED NEURAL NETWORKS;

LEARNING ALGORITHMS;

EID: 84962440920     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (54)

1. Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13(4):341-379, 2003.
2. Asa Ben-Hur and Jason Weston. A user's guide to support vector machines. Methods in Molecular Biology, 609:223-239, 2010.
3. Yoshua Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1):1-127, 2009. Also published as a book. Now Publishers, 2009.
4. Yoshua Bengio. Evolving culture vs local minima. In Growing Adaptive Machines: Integrating Development and Learning in Artificial Neural Networks, number also as ArXiv 1203.2990v1, pages T. Kowaliw, N. Bredeche & R. Doursat, eds. Springer-Verlag, March 2013a. URL http://arxiv.org/abs/1203.2990.
5. Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures. In K.-R. Müller, G. Montavon, and G. B. Orr, editors, Neural Networks: Tricks of the Trade. Springer, 2013b.
6. Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Bernhard Schölkopf, John Platt, and Thomas Hoffman, editors, Ad-vances in Neural Information Processing Systems 19 (NIPS'06), pages 153-160. MIT Press, 2007.
7. Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Léon Bottou and Michael Littman, editors, Proceedings of the Twenty-sixth International Conference on Machine Learning (ICML'09). ACM, 2009.
8. Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. Technical Report arXiv:1206.5538, U. Montreal, 2012. URL http://arxiv.org/abs/1206.5538.
9. Yoshua Bengio, Aaron Courville, and Pascal Vincent. Unsupervised feature learning and deep learning: A review and new perspectives. IEEE Trans. Pattern Analysis and Machine Intel-ligence (PAMI), 2013.
10. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), 2010.
11. Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010, pages 177-186. Springer, 2010.
12. Leo Breiman. Random forests. Machine Learning, 45(1):5-32, 2001.
13. Leo Breiman, Jerome Friedman, Charles J. Stone, and Richard A. Olshen. classification and regression trees. Belmont, Calif.: Wadsworth, 1984.
14. Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surface of multilayer networks. AISTATS 2015, Proceedings of the Eighteenth In-ternational Conference on Artificial Intelligence and Statistics, pages 192-204, 2015.
15. Dan C. Ciresan, Ueli Meier, Luca M. Gambardella, and Jürgen Schmidhuber. Deep big simple neural nets for handwritten digit recognition. Neural Computation, 22:1-14, 2010.
16. Yann Dauphin and Yoshua Bengio. Big neural networks waste capacity. Technical Report arXiv:1301.3583, Universite de Montreal, 2013.
17. Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In NIPS'2014, 2014.
18. Richard Dawkins. The Selfish Gene. Oxford University Press, 1976.
19. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 2011.
20. Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? In Journal of Machine Learning Research JML (-1), pages 625-660.
21. Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel et al Fabian Pedregosa, Gal Varoquaux. Scikit-learn: Machine learning in python. The Journal of Machine Learning Research, 12:2825-2830, 2011.
22. Franois Fleuret, Ting Li, Charles Dubout, Emma K. Wampler, Steven Yantis, and Donald Geman. Comparing machines and humans on a visual categorization test. Proceedings of the National Academy of Sciences, 108(43):17621-17625, 2011.
23. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In JMLR W&CP: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2010), volume 9, pages 249-256, May 2010.
24. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In JMLR W&CP: Proceedings of the Fourteenth International Conference on Artificial Intelli-gence and Statistics (AISTATS 2011), April 2011.
25. Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In ICML'2013, 2013.
26. Caglar Gulcehre, Kyunghyun Cho, Razvan Pascanu, and Yoshua Bengio. Learned-norm pooling for deep neural networks. arXiv preprint arXiv:1311.1780, 2013.
27. Joseph Henrich and Richard McElreath. The evolution of cultural evolution. Evolutionary Anthropology: Issues, News, and Reviews, 12(3):123-135, 2003.
28. Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554, 2006.
29. Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. Technical report, arXiv:1207.0580, 2012.
30. Chih-Wei Hsu, Chih-Chung Chang, and Chih-Jen Lin. A practical guide to support vector classification, 2003.
31. Kevin Jarrett, Koray Kavukcuoglu, Marc'Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In Proc. International Conference on Computer Vision (ICCV'09), pages 2146-2153. IEEE, 2009.
32. Faisal Khan, Xiaojin Zhu, and Bilge Mutlu. How do humans teach: On curriculum learning and teaching dimension. In Advances in Neural Information Processing Systems 24 (NIPS'11), pages 1449-1457, 2011.
33. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS'2012). 2012.
34. Kai A. Krueger and Peter Dayan. Flexible shaping: how learning in small steps helps. Cognition, 110:380-394, 2009.
35. Gautam Kunapuli, Kristin P. Bennett, Richard Maclin, and Jude W. Shavlik. The adviceptron: Giving advice to the perceptron. Proceedings of the Conference on Artificial Neural Networks In Engineering (ANNIE 2010), 2010.
36. Hugo Larochelle, Yoshua Bengio, Jerome Louradour, and Pascal Lamblin. Exploring strategies for training deep neural networks. Journal of Machine Learning Research, 10:1-40, 2009.
37. Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
38. Tom M. Mitchell. The Need for Biases in Learning Generalizations. Department of Computer Science, Laboratory for Computer Science Research, Rutgers Univ., 1980.
39. Tom M. Mitchell and Sebastian B. Thrun. Explanation-based neural network learning for robot control. Advances in Neural information processing systems, pages 287-287, 1993.
40. Richard Montague. Universal grammar. Theoria, 36(3):373-398, 1970.
41. Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 807-814, 2010.
42. Joseph O'Sullivan. Integrating initialization bias and search bias in neural network learning, 1996.
43. Gail B. Peterson. A day of great illumination: B. F. Skinner's discovery of shaping. Journal of the Experimental Analysis of Behavior, 82(3):317-328, 2004.
44. Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the Twenty-eight International Conference on Machine Learning (ICML'11), June 2011.
45. Salah Rifai, Yoshua Bengio, Yann Dauphin, and Pascal Vincent. A generative process for sampling contractive auto-encoders. In Proceedings of the Twenty-nine International Conference on Machine Learning (ICML'12). ACM, 2012. URL http://icml.cc/discuss/2012/590.html.
46. Ruslan Salakhutdinov and Geoffrey E Hinton. Deep boltzmann machines. In International Conference on Artificial Intelligence and Statistics, pages 448-455, 2009.
47. Burrhus F. Skinner. Reinforcement today. American Psychologist, 13:94-99, 1958.
48. Ray J. Solomonoff. A system for incremental learning based on algorithmic probability. In Proceedings of the Sixth Israeli Conference on Artificial Intelligence, Computer Vision and Pattern Recognition, pages 515-527. Citeseer, 1989.
49. Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4, 2012.
50. Geoffrey G. Towell and Jude W. Shavlik. Knowledge-based Artificial neural networks. Artificial intelligence, 70(1):119-165, 1994.
51. Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. In Journal of Machine Learning Research JML (-1), pages 3371-3408.
52. Luis Von Ahn, Manuel Blum, Nicholas J Hopper, and John Langford. Captcha: Using hard ai problems for security. In Advances in Cryptology EUROCRYPT 2003, pages 294-311. Springer, 2003.
53. Jason Weston, Frédéric Ratle, and Ronan Collobert. Deep learning via semi-supervised embedding. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, Proceedings of the Twenty-fifth International Conference on Machine Learning (ICML'08), pages 1168-1175, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-205-4. doi: 10.1145/1390156.1390303.
54. Matthew D Zeiler. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.