Exploiting Hessian matrix and trust-region algorithm in hyperparameters estimation of Gaussian process

Applied Mathematics and Computation

Volumn 171, Issue 2, 2005, Pages 1264-1281

  Zhang, Yunong ^a   Leithead, W E ^a,b  

^a UNIVERSITY OF STRATHCLYDE   (United Kingdom)

^b NATIONAL UNIVERSITY OF IRELAND   (Ireland)

Author keywords

Conjugate gradient; Gaussian process; Hessian matrix; Log likelihood maximization; Trust region

Indexed keywords

ALGORITHMS; MATHEMATICAL MODELS; OPTIMIZATION; PARAMETER ESTIMATION; REGRESSION ANALYSIS;

HESSIAN MATRIX; NON-PARAMETRIC REGRESSION MODEL;

MATRIX ALGEBRA;

EID: 31144441548     PISSN: 00963003     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.amc.2005.01.113     Document Type: Article

References (28)

1. K. Hornik, M. Stinchcombe, and H. White Multilayer feedforward networks are universal approximators Neural Networks 2 1989 259 366
2. E.B. Baum, and D. Haussler What size net gives valid generalization? Neural Computation 6 1989 151 160
3. R.M. Neal Bayesian learning for neural networks Lecture notes in statistics 118 1996 Springer New York
4. C.E. Rasmussen, Evaluation of Gaussian processes and other methods for non-linear regression, Ph.D. thesis, University of Toronto, 1996.
5. D.J.C. MacKay Introduction to Gaussian processes Neural networks and machine learning, F: Computer and Systems Sciences vol. 168 1998 Springer Berlin, Heidelberg pp. 133-165
6. C.K.I. Williams, and D. Barber Bayesian classification with Gaussian processes IEEE Transactions on Pattern Analysis and Machine Intelligence 20 1998 1342 1351
7. S. Sambu, M. Wallat, T. Graepel, K. Obermayer, Gaussian process regression: active data selection and test point rejection, in: Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks, 3, 2000, pp. 241-246.
8. T. Yoshioka, S. Ishii, Fast Gaussian process regression using representative data, in: Proceedings of International Joint Conference on Neural Networks, 1, 2001, pp. 132-137.
9. D.J. Leith, W.E. Leithead, E. Solak, R. Murray-Smith, Divide and conquer identification using Gaussian process priors, in: Proceedings of the 41st IEEE Conference on Decision and Control, 1, 2002, pp. 624-629.
10. J.Q. Shi, R. Murray-Smith, and D.M. Titterington Bayesian regression and classification using mixtures of multiple Gaussian processes International Journal of Adaptive Control and Signal Processing 17 2003 149 161
11. E. Solak, R. Murray-Smith, W.E. Leithead, D.J. Leith, and C.E. Rasmussen Derivative observations in Gaussian process models of dynamic systems Advances in Neural Information Processing Systems 15 2003 1033 1040
12. W.E. Leithead, E. Solak, D.J. Leith, Direct identification of nonlinear structure using Gaussian process prior models, in: European Control Conference, Cambridge, 2003.
13. A. O'Hagan On curve fitting and optimal design for regression Journal of the Royal Statistical Society B 40 1978 1 42
14. K.V. Mardia, and R.J. Marshall Maximum likelihood estimation for models of residual covariance in spatial regression Biometrika 71 1984 135 146
15. N.A.C. Cressie Statistics for spatial data 1993 John Wiley and Sons New York
16. C.J. Paciorek, and M.J. Schervish Nonstationary covariance functions for Gaussian process regression Advances in Neural Information Processing Systems 16 2003
17. J. Nocedal Theory of algorithms for unconstrained optimization Acta Numerica 1992 199 242
18. M.F. Moller A scaled conjugate gradient algorithm for fast supervised learning Neural Networks 6 1993 525 533
19. G.H. Golub, and C.F. Van Loan Matrix computations 1996 Johns Hopkins University Press Baltimore
20. J. Skilling Bayesian numerical analysis, physics and probability 1993 Cambridge University Press
21. Z. Wu, G.N. Phillips Jr., R. Tapia, and Y. Zhang A fast Newton method for entropy maximization in statistical phase estimation Acta Crystallographica A 57 2001 681 685
22. N. Alexandrov, J.E. Dennis Jr., R.M. Lewis, and V. Torczon A trust region framework for managing the use of approximation models in optimization Journal on Structural Optimization 15 1998 16 23
23. J.E. Dennis, and R.B. Schnabel Numerical methods for unconstrained optimization and nonlinear equations 1983 Prentice-Hall Englewood, NJ
24. J.J. More, and D.C. Sorensen Computing a trust region step SIAM Journal of Scientific and Statistically Computing 4 1983 553 572
25. T. Steihaug The conjugate gradient method and trust regions in large scale optimization SIAM Journal on Numerical Analysis 20 1983 626 637
26. R.H. Byrd, R.B. Schnabel, and G.A. Shultz Approximate solution of the trust region problem by minimization over two-dimensional subspaces Mathematical Programming 40 1988 247 263
27. A.R. Conn, N.I.M. Gould, and Ph.L. Toint Trust region methods 2000 SIAM Philadelphia
28. The MathWorks Inc., Optimization Toolbox User's Guide, Version 2.1, 2000.