Generalization properties and implicit regularization for multiple passes SGM

33rd International Conference on Machine Learning, ICML 2016

Volumn 5, Issue , 2016, Pages 3474-3494

  Lin, Junhong ^a   Camoriano, Raffaello ^b   Rosasco, Lorenzo ^c  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b UNIVERSITY OF GENOA   (Italy)

^c ISTITUTO ITALIANO DI TECNOLOGIA   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION ALGORITHMS; ARTIFICIAL INTELLIGENCE; GRADIENT METHODS; STOCHASTIC SYSTEMS;

APPROXIMATION PROPERTIES; GENERALIZATION PROPERTIES; LOSS FUNCTIONS; MULTIPLE PASS; NUMERICAL RESULTS; PARAMETERIZED; STEP SIZE; STOCHASTIC GRADIENT METHODS;

LEARNING SYSTEMS;

EID: 84998886271     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. Bartlett, Peter L and Mendelson, Shahar. Rademacher and Gaussian complexities: Risk bounds and structural results. The Journal of Machine Learning Research, 3: 463-482, 2003.
2. Bartlett, Peter L, Bousquet, Olivier, and Mendelson, Shahar. Local rademacher complexities. Annals of Statistics, 33(4): 1497-1537, 2005.
3. Bertsekas, Dimitri P. Nonlinear Programming. Athena scientific, 1999.
4. Bertsekas, Dimitri P. Incremental gradient, subgradient, and proximal methods for convex optimization: A survey. Optimization for Machine Learning, 2010:1-38, 2011.
5. Bousquet, Olivier and Bottou, Leon. The tradeoffs of large scale learning. In Advances in Neural Information Processing Systems, pp. 161-168, 2008.
6. Boyd, Stephen and Mutapcic, Almir. Stochastic subgradient methods. Notes for EE364b, Standford University, Winter 2007.
7. Boyd, Stephen, Xiao, Lin, and Mutapcic, Almir. Subgradient methods. Lecture notes of EE392o, Stanford University, Autumn Quarter 2003.
8. Chang, Chih-Chung and Lin, Chih-Jen. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1-27:27, 2011.
9. Cucker, Felipe and Zhou, Ding-Xuan. Learning Theory: An Approximation Theory Viewpoint, volume 24. Cambridge University Press, 2007.
10. Dieuleveut, Aymeric and Bach, Francis. Non-parametric stochastic approximation with large step sizes. arXiv preprint arXiv: 1408.0361, 2014.
11. Hardt, Moritz, Recht, Benjamin, and Singer, Yoram. Train faster, generalize better: Stability of stochastic gradient descent. arXiv preprint arXiv:1509.01240, 2016.
12. Lin, Junhong, Rosasco, Lorenzo, and Zhou, Ding-Xuan. Iterative regularization for learning with convex loss functions. The Journal of Machine Learning Research, To appear, 2016.
13. Meir, Ron and Zhang, Tong. Generalization error bounds for Bayesian mixture algorithms. The Journal of Machine Learning Research, 4:839-860, 2003.
14. Nemirovski, Arkadi, Juditsky, Anatoli, Lan, Guanghui, and Shapiro, Alexander. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574-1609, 2009.
15. Orabona, Francesco. Simultaneous model selection and optimization through parameter-free stochastic learning. In Advances in Neural Information Processing Systems, pp. 1116-1124, 2014.
16. Rosasco, Lorenzo and Villa, Silvia. Learning with incremental iterative regularization. In Advances in Neural Information Processing Systems, pp. 1621-1629, 2015.
17. Schmidt, Mark, Roux, Nicolas Le, and Bach, Francis. Minimizing finite sums with the stochastic average gradient. arXiv preprint arXiv: 1309.2388, 2013.
18. Shamir, Ohad and Zhang, Tong. Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. In Proceedings of the 30th International Conference on Machine Learning, pp. 71-79, 2013.
19. Steinwart, Ingo and Christmann, Andreas. Support Vector Machines. Springer Science Business Media, 2008.
20. Tarres, Pierre and Yao, Yuan. Online learning as stochastic approximation of regularization paths: Optimality and almost-sure convergence. IEEE Transactions on Information Theory, 60(9):5716-5735, 2014.
21. Ying, Yiming and Pontil, Massimiliano. Online gradient descent learning algorithms. Foundations of Computational Mathematics, 8(5):561-596, 2008.