Large-scale nonconvex stochastic optimization by Doubly Stochastic Successive Convex approximation

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn , Issue , 2017, Pages 4701-4705

  Mokhtari, Aryan ^a   Koppel, Alec ^a   Scutari, Gesualdo ^b   Ribeiro, Alejandro ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b SCHOOL OF INDUSTRIAL ENGINEERING   (United States)

Author keywords

large scale optimization; lasso; Non convex optimization; parallel optimization; stochastic methods

Indexed keywords

EID: 85023764368     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2017.7953048     Document Type: Conference Paper

References (29)

1. V. Vapnik, The nature of statistical learning theory, 2nd ed. springer, 1999.
2. J. Mairal, F. Bach, and J. Ponce, "Task-driven dictionary learning," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 4, pp. 791-804, 2012.
3. J. Ngiam, A. Coates, A. Lahiri, B. Prochnow, Q. V. Le, and A. Y. Ng, "On optimization methods for deep learning," in Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011, pp. 265-272.
4. R. D. Nowak, S. J. Wright et al., "Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems," IEEE Journal of selected topics in signal processing, vol. 1, no. 4, pp. 586-597, 2007.
5. J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, "Robust face recognition via sparse representation," IEEE transactions on pattern analysis and machine intelligence, vol. 31, no. 2, pp. 210-227, 2009.
6. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Advances in neural information processing systems, 2012, pp. 1097-1105.
7. G. Scutari, F. Facchinei, P. Song, D. P. Palomar, and J.-S. Pang, "Decomposition by partial linearization: Parallel optimization of multiagent systems," Signal Processing, IEEE Transactions on, vol. 62, no. 3, pp. 641-656, 2014.
8. F. Facchinei, G. Scutari, and S. Sagratella, "Parallel selective algorithms for nonconvex big data optimization," Signal Processing, IEEE Transactions on, vol. 63, no. 7, pp. 1874-1889, 2015.
9. H. Robbins and S. Monro, "A stochastic approximation method," Ann. Math. Statist., vol. 22, no. 3, pp. 400-407, 09 1951. [Online]. Available: http://dx.doi.org/10.1214/aoms/1177729586
10. Y. Yang, G. Scutari, D. P. Palomar, and M. Pesavento, "A parallel decomposition method for nonconvex stochastic multi-agent optimization problems," IEEE Transactions on Signal Processing, vol. 64, no. 11, pp. 2949-2964, 2015.
11. A. Bordes, L. Bottou, and P. Gallinari, "SGD-QN: Careful quasiNewton stochastic gradient descent," The Journal of Machine Learning Research, vol. 10, pp. 1737-1754, 2009.
12. N. N. Schraudolph, J. Yu, and S. Günter, "A stochastic quasiNewton method for online convex optimization," in International Conference on Artificial Intelligence and Statistics, 2007, pp. 436-443.
13. A. Mokhtari and A. Ribeiro, "RES: Regularized stochastic BFGS algorithm," Signal Processing, IEEE Transactions on, vol. 62, no. 23, pp. 6089-6104, 2014.
14. -, "Global convergence of online limited memory BFGS," Journal of Machine Learning Research, vol. 16, pp. 3151-3181, 2015.
15. P. Tseng and C. O. L. Mangasarian, "Convergence of a block coordinate descent method for nondifferentiable minimization," J. Optim Theory Appl, pp. 475-494, 2001.
16. Z. Q. Luo and P. Tseng, "On the convergence of the coordinate descent method for convex differentiable minimization," Journal of Optimization Theory and Applications, vol. 72, no. 1, pp. 7-35, 1992.
17. A. Beck and L. Tetruashvili, "On the convergence of block coordinate descent type methods," SIAM Journal on Optimization, vol. 23, no. 4, pp. 2037-2060, 2013.
18. C. Scherrer, A. Tewari, M. Halappanavar, and D. Haglin, "Feature clustering for accelerating parallel coordinate descent," in Advances in Neural Information Processing Systems, 2012, pp. 28-36.
19. P. Richtárik and M. Takáč, "Parallel coordinate descent methods for big data optimization," Mathematical Programming, pp. 1-52, 2015.
20. B. Recht, C. Re, S. Wright, and F. Niu, "Hogwild: A lock-free approach to parallelizing stochastic gradient descent," in Advances in Neural Information Processing Systems, 2011, pp. 693-701.
21. A. Mokhtari, A. Koppel, G. Scutari, and A. Ribeiro, "Large-scale nonconvex stochastic optimization by doubly stochastic successive convex approximation," University of Pennsylvania Technical Report, 2016. [Online]. Available: https://fling.seas.upenn.edu/~akoppel/assets/papers/dssc_report.pdf
22. A. Mokhtari, A. Koppel, and A. Ribeiro, "Doubly random parallel stochastic methods for large scale learning," in 2016 American Control Conference (ACC), July 2016, pp. 4847-4852.
23. J. Mairal, F. Bach, J. Ponce, and G. Sapiro, "Online learning for matrix factorization and sparse coding," The Journal of Machine Learning Research, vol. 11, pp. 19-60, 2010.
24. L. Xiao and T. Zhang, "A proximal stochastic gradient method with progressive variance reduction," SIAM Journal on Optimization, vol. 24, no. 4, pp. 2057-2075, 2014.
25. A. Ruszczyński, "Feasible direction methods for stochastic programming problems," Mathematical Programming, vol. 19, no. 1, pp. 220-229, 1980.
26. B. T. Polyak, Introduction to optimization. Optimization Software New York, 1987.
27. S. Ma, "Alternating proximal gradient method for convex minimization," Journal of Scientific Computing, pp. 1-27.
28. A. Beck and M. Teboulle, "A fast iterative shrinkage-thresholding algorithm for linear inverse problems," SIAM journal on imaging sciences, vol. 2, no. 1, pp. 183-202, 2009.
29. A. Mokhtari, A. Koppel, and A. Ribeiro, "A class of parallel doubly stochastic algorithms for large-scale learning," arXiv preprint arXiv:1606.04991, 2016.