Train longer, generalize better: Closing the generalization gap in large batch training of neural networks

Advances in Neural Information Processing Systems

Volumn 2017-December, Issue , 2017, Pages 1732-1742

  Hoffer, Elad ^a   Hubara, Itay ^a   Soudry, Daniel ^a  

^a TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

STOCHASTIC MODELS; STOCHASTIC SYSTEMS;

ADDITIONAL EXPERIMENTS; COMMON PRACTICES; DIFFUSION BEHAVIOR; GENERALIZATION PERFORMANCE; LEARNING MODELS; NOVEL ALGORITHM; STATISTICAL MODELING; STOCHASTIC GRADIENT DESCENT;

DEEP LEARNING;

EID: 85046996830     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (46)

1. Amodei, D., Anubhai, R., Battenberg, E., et al Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015.
2. Bottou, L. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010, pp. 177-186. Springer, 2010.
3. Bouchaud, J. P. and Georges, A. Anomalous diffusion in disordered media: statistical mechanisms, models and physical applications. Physics reports, 195:127-293, 1990.
4. Bouchaud, J. P. and Comtet, A. Anomalous diffusion in random media of any dimensionality. J. Physique, 48: 1445-1450, 1987.
5. Bray, A. J. and Dean, D. S. Statistics of critical points of Gaussian fields on large-dimensional spaces. Physical Review Letters, 98(15):1-5, 2007.
6. Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., and LeCun, Y. The Loss Surfaces of Multilayer Networks. AISTATS15, 38, 2015.
7. Das, D., Avancha, S., Mudigere, D., et al Distributed deep learning using synchronous stochastic gradient descent. arXiv preprint arXiv:1602.06709, 2016.
8. Dauphin, Y., de Vries, H., Chung, J., and Bengio, Y. Rmsprop and equilibrated adaptive learning rates for non-convex optimization. corr abs/1502.04390 (2015).
9. Dauphin, Y., Pascanu, R., and Gulcehre, C. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In NIPS, pp. 1-9, 2014.
10. Dean, J., Corrado, G., Monga, R., et al Large scale distributed deep networks. In NIPS, pp. 1223-1231, 2012.
11. Deng, J., Dong, W., Socher, R., et al ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
12. Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933, 2017.
13. Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121-2159, 2011.
14. Durrett, R. Multidimensional random walks in random environments with subclassical limiting behavior. Communications in Mathematical Physics, 104(1):87-102, 1986.
15. Ge, R., Huang, F., Jin, C., and Yuan, Y. Escaping from saddle points-online stochastic gradient for tensor decomposition. In COLT, pp. 797-842, 2015.
16. Girosi, F., Jones, M., and Poggio, T. Regularization theory and neural networks architectures. Neural computation, 7(2):219-269, 1995.
17. Goyal, P., Dollár, P., Girshick, R., et al Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.
18. Hardt, M., Recht, B., and Singer, Y. Train faster, generalize better: Stability of stochastic gradient descent. ICML, pp. 1-24, 2016.
19. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
20. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
21. Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR, 2017.
22. Kingma, D. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
23. Krizhevsky, A. Learning multiple layers of features from tiny images. 2009.
24. Krizhevsky, A. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014.
25. LeCun, Y., Bottou, L., and Orr, G. Efficient backprop in neural networks: Tricks of the trade (orr, g. and müller, k., eds.). Lecture Notes in Computer Science, 1524, 1998a.
26. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998b.
27. Li, M. Scaling Distributed Machine Learning with System and Algorithm Co-design. PhD thesis, Intel, 2017.
28. Li, M., Zhang, T., Chen, Y., and Smola, A. J. Efficient mini-batch training for stochastic optimization. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 661-670. ACM, 2014.
29. Luong, M.-T., Pham, H., and Manning, C. D. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025, 2015.
30. Marinari, E., Parisi, G., Ruelle, D., and Windey, P. Random Walk in a Random Environment and 1f Noise. Physical Review Letters, 50(1):1223-1225, 1983.
31. Mnih, V., Kavukcuoglu, K., Silver, D., et al Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.
32. Montavon, G., Orr, G., and Müller, K.-R. Neural Networks: Tricks of the Trade. 2 edition, 2012. ISBN 978-3-642-35288-1.
33. Ruder, S. An overview of gradient descent optimization algorithms. CoRR, abs/1609.04747, 2016.
34. Silver, D., Huang, A., Maddison, C. J., et al Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.
35. Simonyan, K. e. a. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
36. Soudry, D., Hoffer, E., and Srebro, N. The Implicit Bias of Gradient Descent on Separable Data. ArXiv e-prints, October 2017.
37. Soudry, D. and Hoffer, E. Exponentially vanishing sub-optimal local minima in multilayer neural networks. arXiv preprint arXiv:1702.05777, 2017.
38. Srivastava, N., and Hinton, G. E., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.
39. Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139-1147, 2013.
40. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818-2826, 2016.
41. Wan, L., Zeiler, M., Zhang, S., LeCun, Y., and Fergus, R. Regularization of neural networks using dropconnect. ICML'13, pp. III-1058-III-1066. JMLR.org, 2013.
42. Wu, Y., Schuster, M., Chen, Z., et al Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144, 2016.
43. You, Y., Gitman, I., and Ginsburg, B. Scaling sgd batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888, 2017.
44. Zagoruyko, K. Wide residual networks. In BMVC, 2016.
45. Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
46. Zhang, S., and Choromanska, A. E., and LeCun, Y. Deep learning with elastic averaging sgd. In NIPS, pp. 685-693, 2015.