Gradient-based optimization of hyperparameters

Neural Computation

Volumn 12, Issue 8, 2000, Pages 1889-1900

  Bengio, Yoshua ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE;

ALGORITHMS; ARTIFICIAL INTELLIGENCE;

EID: 0034241361     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/089976600300015187     Document Type: Article

References (21)

1. Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, AC-19, 716-728.
2. Becker, S., & LeCun, Y. (1989). Improving the convergence of back-propagation learning with second order methods. In D. Touretzky, G. Hinton, & T. Sejnowski (Eds.), Proceedings of the 1988 Connectionist Models Summer School (pp. 29-37). San Mateo, CA: Morgan Kaufmann.
3. Bengio, Y., & Dugas, C. (1999). Learning simple non-stationarities with hyperparameters. Submitted.
4. Bishop, C. (1992). Exact calculation of the Hessian matrix for the multi-layer perceptron. Neural Computation, 4, 494-501.
5. Bottou, L. (1998). Online algorithms and stochastic approximations. In D. Saad (Ed.), Online learning in neural networks. Cambridge: Cambridge University Press.
6. Breiman, L. (1996). Heuristics of instability and stabilization in model selection. Annals of Statistics, 24, 2350-2383.
7. Craven, P., & Wahba, G. (1979). Smoothing noisy data with spline functions. Numerical Mathematics, 31, 377-403.
8. Grandvalet, Y. (1998). Least absolute shrinkage is equivalent to quadratic penalization. In L. Niklasson, M. Boden, & T. Ziemske (Eds.), ICANN'98 (pp. 201-206). Berlin: Springer-Verlag.
9. Guyon, I., Vapnik, V., Boser, B., Bottou, L., & Solla, S. A. (1992). Structural risk minimization for character recognition. In J. Moody, S. Hanson, & R. Lipmann (Eds.), Advances in neural information processing systems, 4 (pp. 471-479). San Mateo, CA: Morgan Kaufmann.
10. Hinton, G. (1987). Learning translation invariant in massively parallel networks. In J. de Bakker, A. Nijman, & P. Treleaven (Eds.), Proceedings of PARLE Conference on Parallel Architectures and Languages Europe (pp. 1-13). Berlin: Springer-Verlag.
11. Hoerl, A., & Kennard, R. (1970). Ridge regression: Biased estimation for non-orthogonal problems. Technometrics, 12, 55-67.
12. Larsen, J., Svarer, C., Andersen, L. N., & Hansen, L. K. (1998). Adaptive regularization in neural networks modeling. In G. B. Orr & K.-R. Muller (Eds.), Neural Networks: Tricks of the Trade (pp. 113-132). Berlin: Springer-Verlag.
13. Latendresse, S. (1999). Utilisation d'hyper-parametres pour la selection de variable. M.Sc. thesis, University of Montreal.
14. LeCun, Y., Denker, J., & Solla, S. (1990). Optimal brain damage. In D. Touretzky (Ed.), Advances in neural information processing systems, 2 (pp. 598-605). San Mateo, CA: Morgan Kaufmann.
15. MacKay D. (1995). Probable networks and plausible predictions - a review of practical Bayesian methods for supervised neural networks. Network: Computation in Neural Systems, 6, 469-505. Available online at: http://www.iro.umontreal.ca/~lisa.
16. Neal, R. (1998). Assessing relevance determination methods using delve. In C. Bishop (Ed.), Neural networks and machine learning (pp. 97-129). Berlin: Springer-Verlag.
17. Poggio, T., Torre, V., & Koch, C. (1985). Computational vision and regularization theory. Nature, 317, 314-319.
18. Rissanen, J. (1990). Stochastic complexity in statistical inquiry. Singapore: World Scientific.
19. Tibshirani, R. (1995). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society B, 58, 267-288.
20. Tikhonov, A., & Arsenin, V. (1977). Solutions of ill-posed problems. Washington, DC: W. H. Winston.
21. Vapnik, V. (1982). Estimation of dependences based on empirical data. Berlin: Springer-Verlag.