Hyperparameter optimization with approximate gradient

33rd International Conference on Machine Learning, ICML 2016

Volumn 2, Issue , 2016, Pages 1150-1159

  Pedregosa, Fabian ^a  

^a UNIVERSITÉ PARIS DAUPHINE   (France)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; REGRESSION ANALYSIS;

APPROXIMATE GRADIENT; EMPIRICAL PERFORMANCE; GRADIENT INFORMATIONS; HYPER-PARAMETER OPTIMIZATIONS; KERNEL RIDGE REGRESSIONS; LOGISTIC REGRESSIONS; REGULARITY CONDITION; STATE-OF-THE-ART METHODS;

LEARNING SYSTEMS;

EID: 84998567518     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (36)

1. Yoshua Bengio. Gradient-based optimization of hyperparameters. Neural computation, 12(8):1889-1900, 2000.
2. James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. The Journal of Machine Learning Research, 13(1), 2012.
3. James S. Bergstra, Rémi Bardenet, Yoshua Bengio. and Balázs Kégl. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems 24. 2011.
4. Eric Brochu, Vlad M Cora, and Nando De Freitas. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv: 1012.2599, 2010.
5. Luca Calatroni, Cao Chung, Juan Carlos De Los Reyes, Carola-Bibiane Schönlieb, and Tuomo Valkonen. Bilevel approaches for learning of variational imaging models. arXiv preprint arXiv: 1505.02120, 2015.
6. Olivier Chapelle, Vladimir Vapnik, Olivier Bousquet, and Sayan Mukherjee. Choosing multiple parameters for support vector machines. Machine learning, 2002.
7. Alexandre d'Aspremont. Smooth optimization with approximate gradient. SIAM Journal on Optimization, 2008.
8. J. C. De los Reyes, Carola-Bibiane Schönlieb, and Tuomo Valkonen. The structure of optimal parameters for image restoration problems. arXiv preprint arXiv:1505.01953, 2015.
9. Charles-Alban Deledalle, Samuel Vaiter, Jalal Fadili, and Gabriel Peyré. Stein unbiased gradient estimator of the risk (SUGAR) for multiple parameter selection. SIAM Journal on imaging Sciences, 2014.
10. Justin Domke. Generic methods for optimization-based modeling. In International Conference on Artificial Intelligence and Statistics, 2012.
11. Chuan-Sheng Foo, Chuong B. Do, and Andrew Y. Ng. Efficient multiple hyperparameter learning for log-linear models. In Advances in Neural Information Processing Systems 20. 2008.
12. Michael Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 2012.
13. Nicholas J Higham. Accuracy and stability of numerical algorithms. 2002.
14. Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems, 2013.
15. Karl Kunisch and Thomas Pock. A bilevel optimization approach for parameter learning in variational models. SIAM Journal on Imaging Sciences, 2013.
16. Alexandre Lacoste, Hugo Larochelle, François Laviolette, and Mario Marchand. Sequential model-based ensemble optimization. arXiv preprint arXiv: 1402.0796, 2014.
17. Jan Larsen, Lars Kai Hansen, Claus Svarer, and M Ohlsson. Design and regularization of neural networks: the optimal use of a validation set. In Proceedings of the IEEE Signal Processing Society Workshop. IEEE, 1996.
18. Jan Larsen, Claus Svarer, Lars Nonboe Andersen, and Lars Kai Hansen. Adaptive regularization in neural network modeling. In Neural Networks: Tricks of the Trade. Springer, 1998.
19. Chih-Jen Lin, Ruby C Weng, and S. Sathiya Keerthi. Trust region newton method for logistic regression. The Journal of Machine Learning Research, 2008.
20. Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 1989.
21. Wei Liu and Yuhong Yang. Parametric or nonparametric? a parametricness index for model selection. The Annals of Statistics, 2011.
22. Dougal Maclaurin, David Duvenaud, and Ryan P. Adams. Gradient-based hyperparameter optimization through reversible learning. In Proceedings of the 32nd International Conference on Machine Learning, July 2015.
23. p. Technometrics, 1973.
24. Yurii Nesterov. Introductory lectures on convex optimization. Springer Science & Business Media, 2004.
25. Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
26. Peter Ochs, René Ranftl, Thomas Brox, and Thomas Pock. Bilevel optimization with nonsmooth lower level problems. In Scale Space and Variational Methods in Computer Vision. Springer.
27. Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in optimization, 2013.
28. Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural computation, 1994.
29. R. Tyrrell Rockafellar and Roger J-B Wets. Variational analysis. Springer Science & Business Media.
30. Mark Schmidt, Nicolas Le Roux, and Francis R Bach. Convergence rates of inexact proximal-gradient methods for convex optimization. In Advances in neural information processing systems, 2011.
31. Mark Schmidt. Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. arXiv preprint arXiv:1309.2388, 2013.
32. Matthias W Seeger. Cross-validation optimization for large scale structured classification kernel methods. The Journal of Machine Learning Research, 2008.
33. Charles M Stein. Estimation of the mean of a multivariate normal distribution. The annals of Statistics, 1981.
34. Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Freeze-thaw bayesian optimization. arXiv preprint arXiv: 1406.3896, 2014.
35. Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 1996.
36. Athanasios Tsanas, Max A Little, Patrick E McSharry, and Lorraine O Ramig. Accurate telemonitoring of parkinson's disease progression by noninvasive speech tests. Biomedical Engineering, IEEE Transactions on, 2010.