The loss surfaces of multilayer networks

Journal of Machine Learning Research

Volumn 38, Issue , 2015, Pages 192-204

  Choromanska, Anna ^a   Henaff, Mikael ^a   Mathieu, Michael ^a   Arous, Gérard Ben ^a   LeCun, Yann ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; HAMILTONIANS; RANDOM VARIABLES; SIMULATED ANNEALING; SPIN GLASS;

LAYERED STRUCTURES; MULTI-LAYER NETWORK; NON-CONVEX LOSS FUNCTION; NON-ZERO PROBABILITY; PARAMETRIZATIONS; RANDOM MATRIX THEORY; SIMPLE MODELING; SPHERICAL SPIN GLASS;

COMPLEX NETWORKS;

EID: 84954310140     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (25)

1. [Amit et al., 1985] Amit, D. J., Gutfreund, H., and Sompolinsky, H. (1985). Spin-glass models of neural networks. Phys. Rev. A, 32:1007-1018.
2. [Auffinger and Ben Arous, 2013] Auffinger, A. and Ben Arous, G. (2013). Complexity of random smooth functions on the high-dimensional sphere. arXw:1110.5872.
3. [Auffinger et al., 2010] Auffinger, A., Ben Arous, G., and Cerny, J. (2010). Random matrices and complexity of spin glasses. arXiv:1003.1129.
4. [Baldi and Hornik, 1989] Baldi, P. and Hornik, K. (1989). Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 2:53-58.
5. [Bottou, 1998] Bottou, L. (1998). Online algorithms and stochastic approximations. In Online Learning and Neural Networks. Cambridge University Press.
6. [Bray and Dean, 2007] Bray, A. J. and Dean, D. S. (2007). The statistics of critical points of Gaussian fields on large-dimensional spaces. Physics Review Letter.
7. [Dauphin et al., 2014] Dauphin, Y., Pascanu, R., Giilcehre, CC., Cho, K., Ganguli, S., and Bengio, Y. (2014). Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In NIPS.
8. [De la Peiia and Gine, 1999] De la Peiia, V. H. and Gine, E. (1999). Decoupling: from dependence to independence: randomly stopped processes, U-statistics and processes, martingales and beyond. Probability and its applications. Springer.
9. [Denil et al., 2013] Denil, M., Shakibi, B., Dinh, L., Ranzato, M., and Freitas, N. D. (2013). Predicting parameters in deep learning. In NIPS.
10. [Denton et al., 2014] Denton, E., Zaremba, W., Bruna, J., LeCun, Y., and Fergus, R. (2014). Exploiting linear structure within convolutional networks for efficient evaluation. In NIPS.
11. [Dotsenko, 1995] Dotsenko, V. (1995). An Introduction to the Theory ofSpin Glasses and Neural Networks. World Scientific Lecture Notes in Physics.
12. [Fyodorov and Williams, 2007] Fyodorov, Y. V. and Williams, I. (2007). Replica symmetry breaking condition exposed by random matrix calculation of landscape complexity. Journal ofStatistical Physics, 129(5-6), 1081-1116.
13. [Goodfellow et al., 2013] Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013). Maxout networks. In ICML.
14. [Hastie et al., 2001] Hastie, T., Tibshirani, R., and Friedman, J. (2001). The Elements of Statistical Learning. Springer Series in Statistics.
15. [Hinton et al., 2012] Hinton, G., Deng, L., Yu, D., Dahl, G., Rahman Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., and Kingsbury, B. (2012). Deep neural networks for acoustic modeling in speech recognition. Signal Processing Magazine.
16. [Krizhevsky et al., 2012] Krizhevsky, A., Sutskever, I., and Hinton, G. (2012). Imagenet classification with deep convolutional neural networks. In NIPS.
17. [LeCun et al., 1998a] LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998a). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86:2278-2324.
18. [LeCun et al., 1998b] LeCun, Y., Bottou, L., Orr, G., and Muller, K. (1998b). Efficient backprop. In Neural Networks: Tricks of the trade. Springer.
19. [Nair and Hinton, 2010] Nair, V. and Hinton, G. (2010). Rectified linear units improve restricted boltzmann machines. In ICML.
20. [Nakanishi and Takayama, 1997] Nakanishi, K. and Takayama, H. (1997). Mean-field theory for a spin-glass model of neural networks: Tap free energy and the paramagnetic to spin-glass transition. Journal of Physics A: Mathematical and General, 30:8085.
21. [Saad, 2009] Saad, D. (2009). On-line learning in neural networks, volume 17. Cambridge University Press.
22. [Saad and Solla, 1995] Saad, D. and Solla, S. A. (1995). Exact solution for on-line learning in multilayer neural networks. Physical Review Letters, 74(21):4337.
23. [Saxe et al., 2014] Saxe, A. M., McClelland, J. L., and Ganguli, S. (2014). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In ICLR.
24. [Weston et al., 2014] Weston, J., Chopra, S., and Adams, K. (2014). #tagspace: Semantic embed-dings from hashtags. In EMNLP.
25. [Wigner, 1958] Wigner, E. P. (1958). On the Distribution of the Roots of Certain Symmetric Matrices. The Annals of Mathematics, 67:325-327.