Online dictionary learning from big data using accelerated stochastic approximation algorithms

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn , Issue , 2014, Pages 16-20

  Slavakis, Konstantinos ^a   Giannakis, Georgios B ^a  

^a UNIVERSITY OF MINNESOTA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION ALGORITHMS; APPROXIMATION THEORY; BIG DATA; OPTIMIZATION; SIGNAL PROCESSING;

DICTIONARY LEARNING; NUMERICAL TESTS; ONLINE DICTIONARY LEARNING; ONLINE LINEAR REGRESSION; ONLINE OPTIMIZATION ALGORITHMS; STOCHASTIC APPROXIMATION ALGORITHMS; STOCHASTIC APPROXIMATIONS; TOTAL LEAST SQUARES;

E-LEARNING;

EID: 84905259578     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2014.6853549     Document Type: Conference Paper

References (34)

1. I. Tošíc and P. Frossard, "Dictionary learning," IEEE Signal Process. Magaz., vol. 28, no. 2, pp. 27-38, Mar. 2011.
2. J. Mairal, F. Bach, J. Ponce, and G. Sapiro, "Online learning for matrix factorization and sparse coding," J. Machine Learning Research, vol. 11, pp. 19-60, Mar. 2010.
3. J. Mairal, F. Bach, and J. Ponce, "Task-driven dictionary learning," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 34, no. 4, pp. 791-804, Apr. 2012.
4. D. L. Donoho, "Compressed sensing," IEEE Trans. Inform. Theory, vol. 52, pp. 1289-1306, 2006.
5. P. A. Forero, K. Rajawat, and G. B. Giannakis, "Semi-supervised dictionary learning for network-wide link load prediction," in Proc. CIP, Baiona: Spain, May 2012.
6. L. Grippo and M. Sciandrone, "On the convergence of the block nonlinear Gauss-Seidel method under convex constraints," Operations Reseach Letters, vol. 26, no. 3, pp. 127-136, 2000.
7. P. Tseng, "Convergence of block coordinate decent method for nondifferentiable minimization," J. of Optimization Theory and Applications, vol. 109, pp. 475-494, June 2001.
8. P. Tseng and S. Yun, "A coordinate gradient descent method for nonsmooth separable minimization," Math. Program., Ser. B, vol. 117, pp. 387-423, 2009.
9. S. J. Wright, "Accelerated block-coordinate relaxation for regularized optimization," SIAM J. Optim., vol. 22, no. 1, pp. 159-186, 2012.
10. Y. Nesterov, "Efficiency of coordinate descent methods on huge-scale optimization problems," SIAM J. Optim., vol. 22, no. 2, pp. 341-362, 2012.
11. Y. Xu and W. Yin, "A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion," SIAM J. Imaging, vol. 6, no. 3, pp. 1758-1789, 2013.
12. P. Richtarik and M. Takac, "Iteration complexity of randomized blockcoordinate descent methods for minimizing a composite function," Math. Program., Ser. A, pp. 1-38, Dec. 2012.
13. S. J. Wright, R. D. Nowak, and M. A. T. Figueiredo, "Sparse reconstruction by separable approximation," IEEE Trans. Signal Process., vol. 57, no. 7, pp. 2479-2493, July 2009.
14. M. Razaviyayn, M. Hong, and Z.-Q. Luo, "A unified convergence analysis of block successive minimization methods for nonsmooth optimization," SIAM J. Optimization, vol. 23, no. 2, pp. 1126-1153, 2013.
15. S. Shalev-Schwartz and T. Zhang, "Stochastic dual coordinate ascent methods for regularized loss minimization," J. Machine Learning Research, vol. 14, pp. 567-599, 2013.
16. Q. Tao, K. Kong, D. Chu, and G. Wu, "Stochastic coordinate descent methods for regularized smooth and nonsmooth losses," in Lecture Notes in Computer Science, ser. Machine Learning and Knowledge Discovery in Databases, P. A. Flach, T. de Bie, and N. Cristianini, Eds., vol. 7523. Springer, 2012, pp. 537-552.
17. M. Mardani, G. Mateos, and G. B. Giannakis, "Decentralized sparsityregularized rank minimization: Algorithms and applications," IEEE Trans. Signal Process., vol. 61, no. 11, pp. 5374-5388, Nov. 2013.
18. B. Recht and C. Re, "Parallel stochastic gradient algorithms for large-scale matrix completion," 2011, submitted for publication. [Online]. Available: http://pages.cs.wisc.edu/ brecht/papers/11.Rec.Re.IPGM.pdf
19. S. Shalev-Shwartz, "Online learning and online convex optimization," Foundations and Trends in Machine Learning, vol. 4, no. 2, pp. 107-194, 2012.
20. J. Duchi and Y. Singer, "Efficient online and batch learning using forward backward splitting," J. Machine Learning Research, vol. 10, pp. 2899-2934, Dec. 2009.
21. J. Duchi, E. Hazan, and Y. Singer, "Adaptive subgradient methods for online learning and stochastic optimization," J. Machine Learning Research, vol. 12, pp. 2121-2159, 2011.
22. A. Agarwal, S. Negahban, and M. J. Wainwright, "Stochastic optimization and sparse statistical recovery: An optimal algorithm for high dimensions," 2012, submitted for publication. [Online]. Available: ArXiv:1207.4421v1
23. Y. Nesterov, "A method for solving the convex programming problem with convergence rate O(1/k2)," Dokl. Akad. Nauk SSSR, vol. 269, pp. 543-547, 1983, in Russian.
24. A. Beck and M. Teboulle, "A fast iterative shrinkage-thresholding algorithm for linear inverse problems," SIAM J. Imaging Sciences, vol. 2, no. 1, pp. 183-202, 2009.
25. -, "Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems," IEEE Trans. Image Processing, vol. 18, pp. 2419-2439, 2009.
26. M. Yamagishi and I. Yamada, "Over-relaxation of the fast iterative shrinkage-thresholding algorithm with variable stepsize," Inverse Problems, vol. 27, no. 10, 2011.
27. M. Yamagishi, M. Yukawa, and I. Yamada, "Acceleration of adaptive proximal forward-backward splitting method and its application to sparse system identification," in Proc. IEEE ICASSP, Prague: Czech Republic, May 22-27 2011, pp. 4296-4299.
28. C. Hu, J. T. Kwok, and W. Pan, "Accelerated gradient methods for stochastic optimization and online learning," in Proc. of Advances in Neural Information Processing Systems (NIPS), 2009.
29. J. C. Spall, Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control. Wiley, 2003.
30. H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces. New York: Springer, 2011.
31. J. Neveu, Mathematical Foundations of the Calculus of Probability. San Francisco: Holden-Day, 1965.
32. S. Van Huffel and P. Lemmerling, Total Least Squares and Errors-in-Variables Modeling. Springer, 2002.
33. [Online]. Available: http://www.internet2.edu/observatory/
34. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, "Distributed optimization and statistical learning via the alternating direction method of multipliers," Foundations and Trends in Machine Learning, vol. 3, no. 1, pp. 1-122, 2011.