On the influence of momentum acceleration on online learning

Journal of Machine Learning Research

Volumn 17, Issue , 2016, Pages

  Yuan, Kun ^a   Ying, Bicheng ^a   Sayed, Ali H ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Convergence Rate; Heavy ball Method; Mean Square Error Analysis; Momentum Acceleration; Nesterov's Method; Online Learning; Stochastic Gradient

Indexed keywords

GRADIENT METHODS; MEAN SQUARE ERROR; MOMENTUM; OPTIMIZATION; RISK ASSESSMENT; STOCHASTIC SYSTEMS;

CONVERGENCE RATES; HEAVY-BALL METHOD; NESTEROV'S METHODS; ONLINE LEARNING; STOCHASTIC GRADIENT;

E-LEARNING;

EID: 84995388336     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (58)

1. N. O. Attoh-Okine. Analysis of learning rate and momentum term in backpropagation neural network algorithm trained to predict pavement performance. Advances in Engineering Software, 30(4):291-302, 1999.
2. A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183-202, 2009.
3. M. Bellanger. Adaptive Digital Filters and Signal Analysis. 2nd Edition, Marcel Dekker, 2001.
4. D. P. Bertsekas. Nonlinear programming. Athena Scientific, 1999.
5. L. Bottou. Large-scale machine learning with stochastic gradient descent. In Proc. International Conference on Computational Statistics, pages 177-186. Springer, Paris, France, 2010.
6. O. Bousquet and L. Bottou. The tradeoffs of large scale learning. In Proc. Advances in Neural Information Processing Systems, pages 161-168, Vancouver, Canada, 2008.
7. S. Boyd and L. Vandenberghe. Convex optimization. Cambridge University Press, 2004.
8. V. Cevher, S. Becker, and M. Schmidt. Convex optimization for big data: Scalable, randomized, and parallel algorithms for big data analytics. IEEE Signal Processing Magazine, 31(5):32-43, 2014.
9. A. d'Aspremont. Smooth optimization with approximate gradient. SIAM Journal on Optimization, 19(3):1171-1183, 2008.
10. A. Defazio, F. Bach, and S. Lacoste-Julien. SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. In Proc. Advances in Neural Information Processing Systems, pages 1646-1654, Montreal, Canada, 2014.
11. O. Devolder, F. Glineur, and Y. Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming, 146(1-2):37-75, 2014.
12. A. Dieuleveut, N. Flammarion, and F. Bach. Harder, better, faster, stronger convergence rates for least-squares regression. arXiv: 1602.05419, Feb. 2016.
13. J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(2):2121-2159, 2011.
14. N. Flammarion and F. Bach. From averaging to acceleration, there is only a step-size. Journal of Machine Learning Research, 40(1):1-38, 2015.
15. R. Gemulla, E. Nijkamp, P. J. Haas, and Y. Sismanis. Large-scale matrix factorization with distributed stochastic gradient descent. In Proc. International Conference on Knowledge Discovery and Data Mining, pages 69-77, Alberta, Canada, 2011.
16. S. Ghadimi and G. Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization I: A generic algorithmic framework. SIAM Journal on Optimization, 22(4):1469-1492, 2012.
17. S. Haykin. Adaptive Filter Theory. Fourth Edition, Prentice-Hall, NJ, 2008.
18. C. Hu, W. Pan, and J. T. Kwok. Accelerated gradient methods for stochastic optimization and online learning. In Proc. Advances in Neural Information Processing Systems, pages 781-789, Vancouver, Canada, 2009.
19. R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 315-323, Lake Tahoe, Navada, 2013.
20. S. Kahou, C. Pal, X. Bouthillier, P. Froumenty, and et al. Combining modality specific deep neural networks for emotion recognition in video. In Proc. International Conference on Multimodal Interaction, pages 543-550, Sydney, Australia, 2013.
21. A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convo-lutional neural networks. In Proc. Advances in Neural Information Processing Systems, pages 1097-1105, 2012.
22. G. Lan. An optimal method for stochastic composite optimization. Mathematical Programming, 133(1-2):365-397, 2012.
23. L. Lessard, B. Recht, and A. Packard. Analysis and design of optimization algorithms via integral quadratic constraints. SIAM Journal on Optimization, 26(1):57-95, 2016.
24. A. Nedić and D. P. Bertsekas. Convergence rate of incremental subgradient algorithms. In S. Uryasey and M. Pardalos P, editors, Stochastic Optimization: Algorithms and Applications, volume 54, pages 223-264. Springer, 2001.
25. Y. Nesterov. A method of solving a convex programming problem with convergence rate O(1/k2). Soviet Mathematics Doklady, 27(2):372-376, 1983.
26. Y. Nesterov. Introductory Lectures on Convex Optimization. Springer, 2004.
27. Y. Nesterov. Smooth minimization of non-smooth functions. Mathematical Programming, 103(1):127-152, 2005.
28. A. Nitanda. Stochastic proximal gradient descent with acceleration techniques. In Proc. Advances in Neural Information Processing Systems, pages 1574-1582, Montreal, Canada, 2014.
29. B. T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
30. B. T. Polyak. Introduction to Optimization. Optimization Software, NY, 1987.
31. J. G. Proakis. Channel identification for high speed digital communications. IEEE Transactions on Automatic Control, 19(6):916-922, 1974.
32. N. Qian. On the momentum term in gradient descent learning algorithms. Neural Networks, 12(1):145-151, 1999.
33. N. L. Roux, M. Schmidt, and F. R. Bach. A stochastic gradient method with an exponential convergence rate for finite training sets. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 2663-2671, Lake Tahoe, Navada, 2012.
34. S. Roy and J. J. Shynk. Analysis of the momentum LMS algorithm. IEEE Transactions on Acoustics, Speech and Signal Processing, 38(12):2088-2098, 1990.
35. A. H. Sayed. Adaptive Filters. Wiley, NY, 2008.
36. A. H. Sayed. Adaptation, learning, and optimization over networks. Foundations and Trends in Machine Learning, 7(4-5):311-801, Jul. 2014a.
37. A. H. Sayed. Adaptive networks. Proceedings of the IEEE, 102(4):460-497, 2014b.
38. S. Shalev-Shwartz. SDCA without duality. arXiv:1502.06177, Feb. 2015.
39. S. Shalev-Shwartz and T. Zhang. Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization. In Proc. International Conference on Machine Learning, pages 64-72, Beijing, China, 2014.
40. R. Sharma, W. A. Sethares, and J. A. Bucklew. Analysis of momentum adaptive filtering algorithms. IEEE Transactions on Signal Processing, 46(5):1430-1434, 1998.
41. J. J. Shynk and S. Roy. The LMS algorithm with momentum updating. In Proc. IEEE International Symposium on Circuits and Systems, pages 2651-2654, Espoo, Finland, June 1988.
42. I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In Proc. International Conference on Machine Learning, pages 1139-1147, Atlanta, USA, 2013.
43. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelo, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 1-9, Boston, USA, June 2015.
44. S. Theodoridis. Machine Learning: A Bayesian and Optimization Perspective. Academic Press, NY, 2015.
45. L. K. Ting, C. F. N. Cowan, and R. F. Woods. Tracking performance of momentum LMS algorithm for a chirped sinusoidal signal. In Proc. European Signal Processing Conference, pages 1-4, Tampere, Finland, 2000.
46. M. A. Tugay and Y. Tanik. Properties of the momentum LMS algorithm. Signal Processing, 18(2):117-127, 1989.
47. M. Tygert. Poor starting points in machine learning. arXiv:1602.02823, Feb. 2016.
48. B. Widrow and S. D. Stearns. Adaptive Signal Processing. Prentice-Hall, NJ, 1985.
49. W. Wiegerinck, A. Komoda, and T. Heskes. Stochastic dynamics of learning with momentum in neural networks. Journal of Physics A: Mathematical and General, 27(13): 4425-4438, 1994.
50. L. Xiao. Dual averaging methods for regularized stochastic learning and online optimization. Journal of Machine Learning Research, 11(Oct):2543-2596, 2010.
51. B. Ying and A. H. Sayed. Performance limits of online stochastic sub-gradient learning. arXiv:1511.07902, Oct. 2015.
52. B. Ying and A. H. Sayed. Performance limits of single-agent and multi-agent sub-gradient stochastic learning. In Proc. International Conference on Acoustics, Speech and Signal Processing, pages 4905-4909, Shanghai, China, March 2016.
53. K. Yuan, B. Ying, and A. H. Sayed. On the influence of momentum acceleration on online learning. In Proc. International Conference on Acoustics, Speech and Signal Processing, pages 4915-4919, Shanghai, China, March 2016.
54. Ś. Zareba, A. Gonczarek, J. M. Tomczak, and J. Swiatek. Accelerated learning for restricted Boltzmann machine with momentum term. In Proc. International Conference on Systems Engineering, pages 187-192, Coventry, UK, 2015.
55. T. Zhang. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proc. International Conference on Machine Learning, page 116, Alberta, Canada, 2004.
56. X. Zhang and Y. LeCun. Text understanding from scratch. arXiv:1502.01710, Feb. 2015.
57. W. Zhong and J. T. Kwok. Accelerated stochastic gradient method for composite regular-ization. In Proc. International Conference on Artificial Intelligence and Statistics, pages 1086-1094, Reykjavik, Iceland, 2014.
58. Z. Zhu. Katyusha: Accelerated variance reduction for faster SGD. arXiv:1603.05953, Mar. 2016.