Gaussian process dynamic programming

Neurocomputing

Volumn 72, Issue 7-9, 2009, Pages 1508-1524

  Deisenroth, Marc Peter ^a,b   Rasmussen, Carl Edward ^a,c   Peters, Jan ^c,d  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b UNIVERSITY OF KARLSRUHE   (Germany)

^c MAX PLANCK INSTITUTE FOR BIOLOGICAL CYBERNETICS   (Germany)

^d UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

Bayesian active learning; Dynamic programming; Gaussian processes; Optimal control; Policy learning; Reinforcement learning

Indexed keywords

A-PRIORI; APPROXIMATE VALUE FUNCTIONS; APPROXIMATION TECHNIQUES; BAYESIAN ACTIVE LEARNING; CONTINUOUS STATE; GAUSSIAN PROCESSES; INITIAL STATE; ON THE FLIES; OPTIMAL CONTROL; OPTIMAL CONTROL PROBLEMS; POLICY LEARNING; PRIOR KNOWLEDGE; PROBABILISTIC MODELS; STATE SPACES; SWING-UP; TRANSITION DYNAMICS; UNKNOWN VALUES; VALUE FUNCTIONS;

BAYESIAN NETWORKS; DYNAMIC PROGRAMMING; EDUCATION; GAUSSIAN DISTRIBUTION; GAUSSIAN NOISE (ELECTRONIC); OPTIMAL CONTROL SYSTEMS; OPTIMIZATION; REINFORCEMENT; REINFORCEMENT LEARNING; SYSTEMS ENGINEERING; TRELLIS CODES;

PROCESS CONTROL;

ARTICLE; ARTIFICIAL INTELLIGENCE; BAYESIAN LEARNING; CONTROL SYSTEM; CONTROLLED STUDY; GAUSSIAN PROCESS DYNAMIC PROGRAMMING ALGORITHM; INFORMATION PROCESSING; LEARNING ALGORITHM; MACHINE LEARNING; ONLINE SYSTEM; PRIORITY JOURNAL; PROBABILITY; PROCESS OPTIMIZATION;

EID: 61849173491     PISSN: 09252312     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neucom.2008.12.019     Document Type: Article

References (56)

1. C.G. Atkeson, Using local trajectory optimizers to speed up global optimization in dynamic programming, in: J.E. Hanson, S.J. Moody, R.P. Lippmann (Eds.), Advances in Neural Information Processing Systems, vol. 6, Morgan Kaufmann, Los Altos, CA, 1994, pp. 503-521.
2. C.G. Atkeson, J.C. Santamaría, A comparison of direct and model-based reinforcement learning, in: Proceedings of the International Conference on Robotics and Automation, 1997.
3. Atkeson C.G., and Schaal S. Robot learning from demonstration. In: Fisher Jr. D.H. (Ed). Proceedings of the 14th International Conference on Machine Learning (July 1997), Morgan Kaufmann, Nashville, TN, USA 12-20
4. Bellman R.E. Dynamic Programming (1957), Princeton University Press, Princeton, NJ, USA
5. D.P. Bertsekas, Dynamic Programming and Optimal Control, Optimization and Computation Series, vol. 1, third ed., Athena Scientific, Belmont, MA, USA, 2005.
6. D.P. Bertsekas, Dynamic Programming and Optimal Control, Optimization and Computation Series, vol. 2, third ed., Athena Scientific, Belmont, MA, USA, 2007.
7. Bertsekas D.P., and Tsitsiklis J.N. Neuro-Dynamic Programming. Optimization and Computation (1996), Athena Scientific, Belmont, MA, USA
8. Bryson A.E., and Ho Y.-C. Applied Optimal Control: Optimization, Estimation, and Control (1975), Hemisphere, New York City, NY, USA
9. Chaloner K., and Verdinelli I. Bayesian experimental design: a review. Statistical Science 10 (1995) 273-304
10. M.P. Deisenroth, J. Peters, C.E. Rasmussen, Approximate dynamic programming with gaussian processes, in: Proceedings of the 2008 American Control Conference, Seattle, WA, USA, June 2008, pp. 4480-4485.
11. M.P. Deisenroth, C.E. Rasmussen, J. Peters, Model-based reinforcement learning with continuous states and actions, in: Proceedings of the 16th European Symposium on Artificial Neural Networks, Bruges, Belgium, April 2008, pp. 19-24.
12. Doya K. Reinforcement learning in continuous time and space. Neural Computation 12 1 (2000) 219-245
13. Y. Engel, S. Mannor, R. Meir, Bayes meets Bellman: the Gaussian process approach to temporal difference learning, in: Proceedings of the 20th International Conference on Machine Learning, Washington, DC, USA, vol. 20, August 2003, pp. 154-161.
14. Y. Engel, S. Mannor, R. Meir, Reinforcement learning with Gaussian processes, in: Proceedings of the 22nd International Conference on Machine Learning, Bonn, Germany, vol. 22, August 2005, pp. 201-208.
15. Ernst D., Geurts P., and Wehenkel L. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research 6 (2005) 503-556
16. Ghavamzadeh M., and Engel Y. Bayesian policy gradient algorithms. In: Schölkopf B., Platt J.C., and Hoffman T. (Eds). Advances in Neural Information Processing Systems, vol. 19 (2007), The MIT Press, Cambridge, MA, USA 457-464
17. Girard A., Rasmussen C.E., Quiñonero Candela J., and Murray-Smith R. Gaussian process priors with uncertain inputs-application to multiple-step ahead time series forecasting. In: Becker S., Thrun S., and Obermayer K. (Eds). Advances in Neural Information Processing Systems, vol. 15 (2003), The MIT Press, Cambridge, MA, USA 529-536
18. Gordon G.J. Stable function approximation in dynamic programming. In: Prieditis A., and Russell S. (Eds). Proceedings of the 12th International Conference on Machine Learning (1995), Morgan Kaufmann, San Francisco, CA, USA 261-268
19. Howard R.A. Dynamic Programming and Markov Processes (1960), The MIT Press, Cambridge, MA, USA
20. Jacobs R.A., Jordan M.I., Nowlan S.J., and Hinton G.E. Adaptive mixtures of local experts. Neural Computation 3 (1991) 79-87
21. Julier S.J., and Uhlmann J.K. Unscented filtering and nonlinear estimation. IEEE Review 92 3 (2004) 401-422
22. Kalman R.E. A new approach to linear filtering and prediction problems. Transactions of the ASME-Journal of Basic Engineering 82 Series D (1960) 35-45
23. J. Ko, D. Fox, GP-BayesFilters: Bayesian filtering using Gaussian process prediction and observation models, in: Proceedings of the 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Nice, France, September 2008, pp. 3471-3476.
24. J. Ko, D.J. Klein, D. Fox, D. Haehnel, Gaussian processes and reinforcement learning for identification and control of an autonomous blimp, in: Proceedings of the International Conference on Robotics and Automation, Rome, Italy, April 2007, pp. 742-747.
25. J. Ko, D.J. Klein, D. Fox, D. Haehnel, GP-UKF: unscented Kalman filters with Gaussian process prediction and observation models, in: Proceedings of the 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA, USA, October 2007, pp. 1901-1907.
26. J. Kocijan, R. Murray-Smith, C.E. Rasmussen, B. Likar, Predictive control with Gaussian process models, in: B. Zajc, M. Tkalčič (Eds.), Proceedings of IEEE Region 8 Eurocon 2003: Computer as a Tool, Piscataway, NJ, USA, September 2003, pp. 352-356.
27. Krause A., Singh A., and Guestrin C. Near-optimal sensor placements in Gaussian processes: theory, efficient algorithms and empirical studies. Journal of Machine Learning Research 9 (2008) 235-284
28. M. Kuss, Gaussian process models for robust regression, classification, and reinforcement learning, Ph.D. Thesis, Technische Universität Darmstadt, Germany, February 2006.
29. MacKay D.J.C. Information-based objective functions for active data selection. Neural Computation 4 (1992) 590-604
30. MacKay D.J.C. Comparison of approximate methods for handling hyperparameters. Neural Computation 11 5 (1999) 1035-1068
31. MacKay D.J.C. Information Theory, Inference, and Learning Algorithms (2003), Cambridge University Press, The Edinburgh Building, Cambridge, UK
32. R. Martinez-Cantin, N. de Freitas, A. Doucet, J. Castellanos, Active policy learning for robot planning and exploration under uncertainty, in: Proceedings of Robotics: Science and Systems III, Atlanta, GA, USA, June 2007.
33. Matheron G. The intrinsic random functions and their applications. Advances in Applied Probability 5 (1973) 439-468
34. T.P. Minka, A family of algorithms for approximate Bayesian inference, Ph.D. Thesis, Massachusetts Institute of Technology, Cambridge, MA, USA, January 2001.
35. Murray-Smith R., and Sbarbaro D. Nonlinear adaptive control using non-parametric Gaussian process prior models. Proceedings of the 15th IFAC World Congress vol. 15 (July 2002), Academic Press, Barcelona, Spain
36. R. Murray-Smith, D. Sbarbaro, C.E. Rasmussen, A. Girard, Adaptive, cautious, predictive control with Gaussian process priors, in: 13th IFAC Symposium on System Identification, Rotterdam, Netherlands, August 2003.
37. O'Hagan A. Bayes-Hermite quadrature. Journal of Statistical Planning and Inference 29 (1991) 245-260
38. Ormoneit D., and Sen S. Kernel-based reinforcement learning. Machine Learning 49 2-3 (2002) 161-178
39. Peters J., and Schaal S. Learning to control in operational space. The International Journal of Robotics Research 27 2 (2008) 197-212
40. Peters J., and Schaal S. Natural actor-critic. Neurocomputing 71 7-9 (2008) 1180-1190
41. Peters J., and Schaal S. Reinforcement learning of motor skills with policy gradients. Neural Networks 21 (2008) 682-697
42. T. Pfingsten, Bayesian active learning for sensitivity analysis, in: Proceedings of the 17th European Conference on Machine Learning, September 2006, pp. 353-364.
43. Quiñonero-Candela J., and Rasmussen C.E. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research 6 2 (2005) 1939-1960
44. C.E. Rasmussen, Evaluation of Gaussian processes and other methods for non-linear regression, Ph.D. Thesis, Department of Computer Science, University of Toronto, 1996.
45. C.E. Rasmussen, M.P. Deisenroth, Probabilistic inference for fast learning in control, in: S. Girgin, M. Loth, R. Munos, P. Preux, D. Ryabko (Eds.), Recent Advances in Reinforcement Learning, Lecture Notes on Computer Science, vol. 5323, Springer, Berlin, November 2008, pp. 229-242.
46. Rasmussen C.E., and Ghahramani Z. Bayesian Monte Carlo. In: Becker S., Thrun S., and Obermayer K. (Eds). Advances in Neural Information Processing Systems, vol. 15 (2003), The MIT Press, Cambridge, MA, USA 489-496
47. Rasmussen C.E., and Kuss M. Gaussian processes in reinforcement learning. In: Thrun S., Saul L.K., and Schölkopf B. (Eds). Advances in Neural Information Processing Systems, vol. 16 (2004), The MIT Press, Cambridge, MA, USA 751-759
48. Rasmussen C.E., and Williams C.K.I. Gaussian processes for machine learning. Adaptive Computation and Machine Learning (2006), The MIT Press, Cambridge, MA, USA. http://www.gaussianprocess.org/gpml/ URL 〈http://www.gaussianprocess.org/gpml/〉
49. Riedmiller M. Concepts and facilities of a neural reinforcement learning control architecture for technical process control. Neural Computation and Application 8 (2000) 323-338
50. M. Riedmiller, Neural Fitted Q iteration-first experiences with a data efficient neural reinforcement learning method, in: Proceedings of the 16th European Conference on Machine Learning, Porto, Portugal, 2005.
51. Riedmiller M., and Braun H. A direct adaptive method for faster backpropagation learning the RPROP algorithm. Proceedings of the IEEE International Conference on Neural Networks (1993) 586-591
52. Snelson E., and Ghahramani Z. Sparse Gaussian processes using pseudo-inputs. In: Weiss Y., Schölkopf B., and Platt J.C. (Eds). Advances in Neural Information Processing Systems, vol. 18 (2006), The MIT Press, Cambridge, MA, USA 1257-1264
53. Sutton R.S., and Barto A.G. Reinforcement Learning An Introduction. Adaptive Computation and Machine Learning (1998), The MIT Press, Cambridge, MA, USA
54. Verdinelli I., and Kadane J.B. Bayesian designs for maximizing information and outcome. Journal of the American Statistical Association 87 418 (1992) 510-515
55. Wasserman L. All of Nonparametric Statistics. Springer Texts in Statistics (2006), Springer Science+Business Media, Inc., New York, NY, USA
56. Williams C.K.I., and Rasmussen C.E. Gaussian processes for regression. In: Touretzky D.S., Mozer M.C., and Hasselmo M.E. (Eds). Advances in Neural Processing Systems, vol. 8 (1996), The MIT Press, Cambridge, MA, USA 598-604